Azure Question posted in
The official documentation to get you started can be found here.

Azure VM run-commands freeze and hang and starts to block DevOps pipeline from running new run-commands on VM with Azure CLI

I’m running couple of PowerShell scripts as part of our DevOps pipelines on Windows Azure VMs with az vm run-command create .. tool. Sometimes the commands freeze and won’t complete in decent time. When this happens I’m not able to execute those commands anymore and it starts to block the devops pipeline runs. …and also if I try to delete the run-command with az vm run-command delete .. it also freezes.

Example of run-command execution (and delete) as part of devops pipeline:

# Execute the script on VM with run-command
az vm run-command create 
    --name RecreateBinariesFolder$(Build.BuildId) 
    --vm-name ${{ parameters.vmName }} 
    --resource-group my-group-${{ parameters.environment }} 
    --timeout-in-seconds 600 
    --script "if(Test-Path C:\temp\packages\){ Remove-Item -Path c:\temp\packages -Force -Recurse } ; if(!(Test-Path C:\temp\packages\)){ mkdir c:\temp\packages }"
# Delete the run-command from VM
az vm run-command delete 
    --name RecreateBinariesFolder$(Build.BuildId) 
    --vm-name ${{ parameters.vmName }} 
    --resource-group my-group-${{ parameters.environment }} 
    --yes

It works well usually on fresh VM but then executions start to freeze after awhile in later pipeline runs.

Any way to execute run-commands in more stabile way or are there any other handy ways to run commands on Windows VMs more easily and in more stabile manner?

Update 2023-01-23: After re-creating the VMs and starting to run the delete command with --no-wait option we haven’t noticed similar freezing anymore after tens of pipeline runs. There was most likely some unexpected issue with the VM itself or the CustomScriptExtension that runs the commands on the VM.

Tags:

2

Answers


  1. 0

    If your requirement is to run a powershell script, why simply don’t you use the default command RunPowerShellScript as described in the third example of az vm run-command invoke documentation?

    The RunPowershellScript is part of available commands for Windows VMs as described in Run scripts in your Windows VM by using action Run Commands.

    This could be the command in your case:

    az vm run-command invoke 
  --command-id RunPowerShellScript 
  --name ${{ parameters.vmName }} 
  -g my-group-${{ parameters.environment }} 
  --scripts 'if(Test-Path C:\temp\packages\){ Remove-Item -Path c:\temp\packages -Force -Recurse } ; if(!(Test-Path C:\temp\packages\)){ mkdir c:\temp\packages }'

    In this way you doesn’t need to create a new command type and remove it.

    In addition, in your code you create a new command but never invoke it. Did you miss the code to invoke it?

    It seems that the maximum number of commands that could be created in a vm is 25 (see docs, so if you try to create more commands probably the system doesn’t work.

    Login or Signup to reply.
  2. 0

    Solution: After re-creating the VMs and starting to run the delete command with --no-wait option, they haven’t noticed similar freezing anymore.

    If you have the same issue, you can try the followings to narrow down the issue.

    1. Use MS-hosted agent to run the pipeline. If it has the same issue, it may be related to the agent or your scripts.
    2. Run the same commands multiple times on the affected VM directly in the Azure portal.
    3. Create a new VM, run the same commands multiple times on it directly in the Azure portal. If the old VM freezes but the new one works, the issue may be related to the old VM itself. If both the old and new VM work, but freezes when used in the pipeline, the issue may be related to the agent.
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search