GitLab Runner suddenly fails to run jobs using Docker Machine and AWS Autoscaling

Update:

In the meantime, GitLab have released a new version of their Docker Machine fork which upgrades the default AMI to Ubuntu 20.04. That means that upgrading Docker Machine to the latest version released by GitLab will fix the issue without changing your runner configuration. The latest release can be found here.

Original Workaround/fix:

Explicitly specify the AMI in your runner configuration and do not rely on the default one anymore, i.e. add something like "amazonec2-ami=ami-02584c1c9d05efa69" to your MachineOptions:

MachineOptions = [
  "amazonec2-access-key=xxx",
  "amazonec2-secret-key=xxx",
  "amazonec2-region=eu-central-1",
  "amazonec2-vpc-id=vpc-xxx",
  "amazonec2-subnet-id=subnet-xxx",
  "amazonec2-use-private-address=true",
  "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true",
  "amazonec2-security-group=ci-runners",
  "amazonec2-instance-type=m5.large",
  "amazonec2-ami=ami-02584c1c9d05efa69",  # Ubuntu 20.04 for amd64 in eu-central-1
  "amazonec2-request-spot-instance=true",
  "amazonec2-spot-price=0.045"
]

You can get a list of Ubuntu AMI IDs here. Be sure to select one that fits your AWS region and instance architecture and is supported by Docker.

Explanation:

The default AMI that GitLab Runner / the Docker Machine EC2 driver use is Ubuntu 16.04. The install script for Docker, which is available on https://get.docker.com/ and which Docker Machine relies on, seems to have stopped supporting Ubuntu 16.04 recently. Thus, the installation of Docker fails on the EC2 instance spawned by Docker Machine and the job cannot run.

See also this GitLab issue.

Azure and GCP suffer from similar problems.