How to view resource allocation of Azure Pipeline agent? – Ubuntu

I am running integration tests in Azure Pipelines. I spin up two Docker containers. One container holds my test project and another container has the Postgres database.

When I run the docker compose on my local machine, the tests run successfully and take about 6 minutes.

When I run the same docker containers in the pipeline, the job doesn’t finish. The job is canceled because of the 60 min limit.

The job running on agent Hosted Agent ran longer than the maximum time of 60 minutes

I do not see any helpful data in the logs.

What tools/logs can I use to diagnose this issue?

It might have to do with RAM or CPU allocation.

Is there a way to do docker stats to see how many resources are allocated to docker containers?

Also, I have multiple test projects and I’m testing them (in the pipeline) one at a time. There are projects that succeeded with this setup. So this approach works, however, when it fails as described, there isn’t a way forward to troubleshoot.

The pipeline:

pool:
  vmImage: ubuntu-latest

stages:
  - stage: Build
    displayName: Docker compose build & up
    jobs:
      - job: Build
        displayName: Build
        steps:
          - script: |
              docker compose build --no-cache
              docker compose up --abort-on-container-exit
            displayName: 'Docker Compose Build & Up'

The docker compose that pipeline calls:

version: "3.8"

services:
  test_service:
    container_name: test_service
    image: test_service_image
    build:
      context: .
      dockerfile: Dockerfile
    environment:
      ASPNETCORE_ENVIRONMENT: Staging
      WAIT_HOSTS: integration_test_db_server:5432
    volumes:
      - ./TestResults:/var/temp
    depends_on:
      - integration_test_db_server
    deploy:
      resources:
        limits:
          memory: 4gb
  
  integration_test_db_server:
    image: postgres
    container_name: db_server
    restart: always
    ports:
      - "2345:5432"
    environment:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
      POSTGRES_DB: db_server

Dockerfile refernced by test_service:

FROM mcr.microsoft.com/dotnet/sdk:6.0

WORKDIR /

COPY . ./

ADD https://github.com/ufoscout/docker-compose-wait/releases/download/2.9.0/wait /wait  
#RUN chmod +x /wait
RUN /bin/bash -c 'ls -la /wait; chmod +x /wait; ls -la /wait'

CMD /wait && dotnet test ./src/MyPorject/MyProject.Tests.csproj  --logger trx --results-directory /var/temp 

UPDATE – Jan 3rd 2023:

I was able to reproduce this on my local machine. Because the MSFT agent is limited to 2 cores, I made that same restriction in the docker-compose file.

This caused a test to run for a very long time (over 8 minutes for one test). At that time, the CPU usage was < 3%.

Running docker stats while the test is running

So restricting the number of CPU cores, causes less CPU usage? I am confused as to what’s happening here.