Solution

In order to test ansible roles with multiple versions of ansible, python and various Linux flavors we can use

• molecule for our ansible role functionality
• docker as our abstraction layer on which we run the target system for our ansible role
• tox to setup generic virtualenvs and test our various combinations without side effects
• travis to automate it all

This will be quite a long/detailed answer. You can check out an example ansible role with the whole setup here

• https://github.com/ckaserer/ansible-role-example
• https://travis-ci.com/ckaserer/ansible-role-example

---

Step 1: test ansible role with molecule

Molecule docs: https://molecule.readthedocs.io/en/stable/

Fixes Issue: 1) no official ansible images

I could create ansible images for every distro I would like to test as jeff geerling describes in his blog posts.

The clear downside of this approach: The images will need maintenance (eventually)

However, with molecule, we can combine base images with a Dockerfile.j2 (Jinja2 Template) to create the images with minimal requirements to run ansible. Through this approach, we are now able to use the official linux distro images from docker hub and do not need to maintain a docker repo for each linux distro and the various versions.

Here the important bits in molecule.yml

platforms:
  - name: instance-${TOX_ENVNAME}
    image: ${MOLECULE_DISTRO:-'centos:7'}
    command: /sbin/init
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    privileged: true

The default dockerfile.j2 from molecule is already good, but I have a few additions

# Molecule managed

{% if item.registry is defined %}
FROM {{ item.registry.url }}/{{ item.image }}
{% else %}
FROM {{ item.image }}
{% endif %}

{% if item.env is defined %}
{% for var, value in item.env.items() %}
{% if value %}
ENV {{ var }} {{ value }}
{% endif %}
{% endfor %}
{% endif %}

RUN if [ $(command -v apt-get) ]; then                                                        apt-get update && apt-get install -y python sudo bash ca-certificates iproute2 && apt-get clean; 
    elif [ $(command -v zypper) ]; then                                                       zypper refresh && zypper install -y python sudo bash python-xml iproute2 && zypper clean -a; 
    elif [ $(command -v apk) ]; then                                                          apk update && apk add --no-cache python sudo bash ca-certificates; 
    elif [ $(command -v xbps-install) ]; then                                                 xbps-install -Syu && xbps-install -y python sudo bash ca-certificates iproute2 && xbps-remove -O; 
    elif [ $(command -v swupd) ]; then                                                        swupd bundle-add python3-basic sudo iproute2; 
    elif [ $(command -v dnf) ] && cat /etc/os-release | grep -q '^NAME=Fedora';          then dnf makecache && dnf --assumeyes install python sudo python-devel python*-dnf bash iproute && dnf clean all; 
    elif [ $(command -v dnf) ] && cat /etc/os-release | grep -q '^NAME="CentOS Linux"' ; then dnf makecache && dnf --assumeyes install python36 sudo platform-python-devel python*-dnf bash iproute && dnf clean all && ln -s /usr/bin/python3 /usr/bin/python; 
    elif [ $(command -v yum) ]; then                                                          yum makecache fast && yum install -y python sudo yum-plugin-ovl bash iproute && sed -i 's/plugins=0/plugins=1/g' /etc/yum.conf && yum clean all; 
    fi

# Centos:8 + ansible 2.7 failed with error: "The module failed to execute correctly, you probably need to set the interpreter"
# Solution: ln -s /usr/bin/python3 /usr/bin/python

By default, this will test the role with centos:7. However, we can set the environment variable MOLECULE_DISTRO to whichever image we would like to test and run it via

MOLECULE_DISTRO=ubuntu:18.04 molecule test

Summary

We use official distro images from docker hub to test our ansible role via molecule.

The files used in this step

• molecule.yml https://github.com/ckaserer/ansible-role-example/blob/master/molecule/default/molecule.yml
• Dockerfile.j2 https://github.com/ckaserer/ansible-role-example/blob/master/molecule/default/Dockerfile.j2

Source

• https://molecule.readthedocs.io/en/stable/examples.html#systemd-container
• https://www.jeffgeerling.com/blog/2019/how-i-test-ansible-configuration-on-7-different-oses-docker
• https://www.jeffgeerling.com/blog/2018/testing-your-ansible-roles-molecule

---

Step 2: Check compatibility for your role ( python version X ansible version X linux distros )

Fixes Issue 2: how to best test a role for ansible version compatibility?

Let's use tox to create virtual environments to avoid side effects while testing various compatibility scenarios.

Here the important bits in tox.ini

[tox]
minversion = 3.7
envlist = py{3}-ansible{latest,29,28}-{   alpinelatest,alpine310,alpine39,alpine38,   centoslatest,centos8,centos7,   debianlatest,debian10,debian9,debian8,   fedoralatest,fedora30,fedora29,fedora28,   ubuntulatest,ubuntu2004,ubuntu1904,ubuntu1804,ubuntu1604   }

# only test currently supported ansible versions
# https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html#release-status

skipsdist = true

[base]
passenv = *
deps =
    -rrequirements.txt
    ansible25: ansible==2.5
    ...
    ansiblelatest: ansible
commands =
    molecule test
setenv =
    TOX_ENVNAME={envname}
    MOLECULE_EPHEMERAL_DIRECTORY=/tmp/{envname}

[testenv]
passenv = 
    {[base]passenv}
deps = 
    {[base]deps}
commands = 
    {[base]commands}
setenv =
    ...
    centoslatest:     MOLECULE_DISTRO="centos:latest"
    centos8:          MOLECULE_DISTRO="centos:8"
    centos7:          MOLECULE_DISTRO="centos:7"
    ...
    {[base]setenv}

The entirety of requirements.txt

docker
molecule

by simply executing

tox

it will create virtual envs for each compatibility combination defined in tox.ini by

envlist = py{3}-ansible{latest,29,28}-{   alpinelatest,alpine310,alpine39,alpine38,   centoslatest,centos8,centos7,   debianlatest,debian10,debian9,debian8,   fedoralatest,fedora30,fedora29,fedora28,   ubuntulatest,ubuntu2004,ubuntu1904,ubuntu1804,ubuntu1604   }

which translates in this particular case to: python3 x ansible version x linux distro

Great! We have created tests for compatibility checks with the added benefit of always testing with ansible latest to notice breaking changes early on.

The files used in this step

• tox.ini https://github.com/ckaserer/ansible-role-example/blob/master/tox.ini
• requirements.txt https://github.com/ckaserer/ansible-role-example/blob/master/requirements.txt

Source

• https://tox.readthedocs.io/en/latest/
• https://molecule.readthedocs.io/en/stable/testing.html#tox

---

Step 3: CI with travis

Running the checks locally is good, running in a CI tool is great. So let's do that.

For this purpose following bits in the .travis.yml are important

---
version: ~> 1.0

os: linux
language: python

python:
  - "3.8"
  - "3.7"
  - "3.6"
  - "3.5"

services: docker

cache:
  pip: true
  directories:
  - .tox

install:
  - pip install tox-travis
env:
  jobs:
    # ansible:latest - check for breaking changes
    ...
    - TOX_DISTRO="centoslatest"     TOX_ANSIBLE="latest"
    - TOX_DISTRO="centos8"          TOX_ANSIBLE="latest"
    - TOX_DISTRO="centos7"          TOX_ANSIBLE="latest"
    ...
    # ansible: check version compatibility
    # only test currently supported ansible versions
    # 
https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html#release-status
    - TOX_DISTRO="centoslatest"     TOX_ANSIBLE="{29,28}"
    - TOX_DISTRO="centos8"          TOX_ANSIBLE="{29,28}"
    - TOX_DISTRO="centos7"          TOX_ANSIBLE="{29,28}"
    ...  
script:
  - tox -e $(echo py${TRAVIS_PYTHON_VERSION} | tr -d .)-ansible${TOX_ANSIBLE}-${TOX_DISTRO}
   # remove logs/pycache before caching .tox folder  
  - |
    rm -r .tox/py*/log/*                                                               
    find . -type f -name '*.py[co]' -delete -o -type d -name __pycache__ -delete

First we have specified language: python to run builds with multiple versions of python as defined in the python: list.

We need docker, so we add it via services: docker.

The test will take quite some time, let's cache pip and our virtenv created by tox with

cache:
  pip: true
  directories:
  - .tox

We need tox...

install:
  - pip install tox-travis

And finally, we define all our test cases

env:
  jobs:
    # ansible:latest - check for breaking changes
    ...
    - TOX_DISTRO="centoslatest"     TOX_ANSIBLE="latest"
    ...

Note that I have separate jobs for latest and the distinct versions. That is on purpose. I would like to easily see what broke. Is it version compatibility or an upcoming change located in ansible's latest release.

The files used in this step

• https://github.com/ckaserer/ansible-role-example/blob/master/.travis.yml

Source

• https://docs.travis-ci.com/user/caching/

---

Bonus: run tox in parallel

You can run the tests in parallel (e.g. 3 test simultaneous) by executing

tox -p 3

However, this will not give the output from molecule. You can enable that with

tox -p 3 -o true

The obvious downside to this approach is the pain of figuring out which line belongs to which process in the parallel execution.

Source

• https://tox.readthedocs.io/en/latest/example/basic.html#parallel-mode