skip to Main Content

I have an Automation Account in Azure, with a few PowerShell Runbooks. The runbooks run on a Windows Hybrid Worker VM also in Azure. All my runbooks except for one run without issue. The problem runbooks runs fine, but then restarts itself, and it does that over and over again.

The relevant code from the runbook is:

Initialize-Variables

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }

$session_token = Get-Token

.log-message.ps1 -RunbookVariables $script:CommonVars -Message "Polling ${script:secret_source} for certificates"

$devices = Get-Devices -session_token $session_token

$certificates = @();
$certificate_types = @("local", "remote", "ca", "pki", "ocsp-server");

foreach ($device in $devices) {
    $device_id = $device.vdom[0].devid;
    $device_certs = @();
    foreach ($cert_type in $certificate_types) {
        $type_certs = Get-DeviceCerts -session_token $session_token -device_id $device_id -cert_type $cert_type
        if ($null -eq $type_certs) { Continue }
        try {
            $device_certs += $type_certs.Where({ $null -ne $_._certinfo })
        }
        catch {
            $device_certs += @($type_certs).Where({ $null -ne $_._certinfo })
        }
    }

    .log-message.ps1 -RunbookVariables $script:CommonVars -Message "Checking '${device_id}' for certificates"
    foreach ($cert in $device_certs) {
        $parsed_cert = Set-Certificate -cert $cert -device_id $device_id;
        # Confirm the cert is not a duplicate
        if (!(Find-Duplicate -certificates $certificates -cert $parsed_cert)) {
            $certificates += @($parsed_cert)
        }
    }
}

Remove-Token -session_token $session_token

.log-message.ps1 -RunbookVariables $script:CommonVars -Message "${script:secret_source} Certificates polled"
.log-message.ps1 -RunbookVariables $script:CommonVars -Message "Checking delta table for diffs"

$script:CommonVars["secrets"] = $certificates

.manage-secret-database.ps1 -RunbookVariables $script:CommonVars
.log-message.ps1 -RunbookVariables $script:CommonVars -Message "FINISHED POLLING CERTIFICATES FOR FORTIGATE"

# Nullify sensitive vars, letting the garbage collector know they're ready to be collected
$script:db_client_secret = $null
$script:db_password = $null
$script:fortimanager_password = $null

# Nullify secret array, letting the garbage collector know it's ready to be collected
# The secret array doesn't contain any sensitive info per-se, but since it's metadata on secrets/certs, it's best to clear it anyway
$script:secrets = $null
$script:certificates = $null

.log-message.ps1 -RunbookVariables $script:CommonVars -Message "Starting Garbage Collection"
# Force Garbage collection
[GC]::Collect()

# FIXME: unless we explicitly write output at the end of this specific script, it will call itself infinitely
Start-Sleep 10
Write-Output "Attempting to Exit"
Exit 0

The script calls a few other runbooks log-message, initialize-common-vars, and manage-secret-database. But the other runbooks call these same runbooks without restarting.
I thought the issue was that the script needed to output something so I added logic to write some output then explicitly exit, but the output gets written, then the runbook just restarts.

I asked ChatGPT, and I searched high & low on Google but couldn’t find a solution. Any ideas?

2

Answers


  1. Chosen as BEST ANSWER

    I was able to resolve this issue. The problem is that, even with a Hybrid Worker, if a Runbook spawns a child runbook, that child only has 10 minutes to execute. If the child runbook fails to finish within 10 minutes, the Parent Runbook is restarted (without killing the child runbook's process)


  2. Remove Start-Sleep 10 and Exit 0 at the end of your script because it might be causing a restart, In azure automation the script execution is considered complete when the script reaches the end . Adding the sleep and exit might be causing the runbook to restart.

    Instead, you can use Start-Sleep and Wait-Event within the script to ensure the delay or wait for a specific condition.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search