Json – A more efficient way of using Ansible to compare two lists of dictionaries, and creating a new structure with elements from both

I have an Ansible playbook that is responsible for the creation of items using an API. To check that what I have just created is working I pull status information, post creation, from another API endpoint. I then need to check the status information against what was just created to ensure that everything is working correctly.

The information gathered about the statuses is stored within another list of dictionaries, which contains information from both lists. The generated information is then checked to see if everything is functionining correcty. I have some code which partly solves this problem, but it is full of loops and not very efficient.

I was therefore wondering if someone could help me with the above?

The information about the things I am creating has been read from a .csv file, the resulting list looks like this (data has been purposely obfuscated and truncated for privacy, but the structure is correct) the output is as per an ansible.builtin.debug output on the variable csv_test_data which stores the contents of a csv file, as read by community.general.read_csv:

ok: [...] => {
  "csv_test_data" : {
    "changed": false,
    "dict": {},
    "failed": false,
    "list": [
      {
        "Service GroupWebservice": "SERVICE_GROUP_01",
        "ServiceGroupWebsite": "SITE_GROUP_01",
        "ServiceIP1": "1.1.1.1",
        "Service1ServerName": "SERVER0101",
        "ServiceIP2": "2.2.2.2",
        "Service2ServerName": "SERVER0102",
        "WebServiceIP1": "10.10.10.10",
        "WebService1ServerName": "WEBSERVER0101",
        "WebServiceIP2": "11.11.11.11",
        "WebService2ServerName": "WEBSERVER0102",
        "EnvironmentName": "TEST",
        "CustomerName": "CUSTOMER01",
        "WebserviceMonitorName": "MON_WEBSERVICE_01",
        "WebsiteMonitorName": "MON_WEBSITE_01"
      },
      {
        "Service GroupWebservice": "SERVICE_GROUP_02",
        "ServiceGroupWebsite": "SITE_GROUP_02",
        "ServiceIP1": "5.5.5.5",
        "Service1ServerName": "SERVER0201",
        "ServiceIP2": "6.6.6.6",
        "Service2ServerName": "SERVER0202",
        "WebServiceIP1": "15.15.15.15",
        "WebService1ServerName": "WEBSERVER0201",
        "WebServiceIP2": "16.16.16.16",
        "WebService2ServerName": "WEBSERVER0202",
        "EnvironmentName": "TEST",
        "CustomerName": "CUSTOMER02",
        "WebserviceMonitorName": "MON_WEBSERVICE_02",
        "WebsiteMonitorName": "MON_WEBSITE_02"
      },
      ...
    ]
  }
}

The status information that I am checking against is contained within another list of dictionaries that is pulled from an API endpoint. This contains status information about all instances of health monitors; i.e. more than what I just created, so I need to compare against a subset of this information. Again the structure is correct and taken directly from an ansible.builtin.debug output on the variable get_reponse_oper, but the data it obfuscated and truncated.

ok: [...] => {
  "get_response_oper": {
    ...
    "changed": false,
    "failed": false,
    "json": {
      "health-stat": {
        ...
        "oper": {
          "health-check-list" [
            {
              ...
              "down-cause": 101,
              "down-state": 10,
              "health-monitor": "default",
              "ip-address": "200.200.200.200",
              "server": "SOMESERVER",
              "status": "DOWN"
            },
            {
              ...
              "down-cause": 7,
              "down-state": 8,
              "health-monitor": "MON_WEBSERVICE_01",
              "ip-address": "1.1.1.1",
              "server": "SERVER0101",
              "status": "UP"
            },
            {
              ...
              "down-cause": 7,
              "down-state": 8,
              "health-monitor": "MON_WEBSERVICE_01",
              "ip-address": "2.2.2.2",
              "server": "SERVER0102",
              "status": "UP"
            },
            {
              ...
              "down-cause": 41,
              "down-state": 10,
              "health-monitor": "MON_WEBSITE_02",
              "ip-address": "15.15.15.15",
              "server": "WEBSERVER0201",
              "status": "DOWN"
            },
            ...
          ]
        }
      }
    }
  }
}

In terms of what I am generating, I want to resulting list of dictionaries to be created, so that I can see/check the status of each of the created items.

In this list there will be one entry per customer, i.e. one for each line within the initial creation .csv file (each customer has four IP addresses, two for servers, two for web servers).

The status of each monitor is taken from the second list (stored in health-check-list), cross-referenced using the "ip-address" and/or "server" which is taken from the first list.

{
  "health_monitor_statuses": [
    {
      "customer": "CUSTOMER01",
      "serviceIp1": "UP",
      "serviceIP2": "UP",
      "site1IP1": "UP",
      "site2IP2": "UP"
    },
    {
      "customer": "CUSTOMER02",
      "serviceIp1": "DOWN",
      "serviceIP2": "DOWN",
      "site1IP1": "DOWN",
      "site2IP2": "DOWN"
    }
  ]
}

At the end I will loop over the created health_monitor_statuses list and fail if I see any "DOWN" or "UNKN" entries.

I believe that is possible/plausible in Ansible. Any help/guidance would be greatly appreciated.

I have tried a number of different approaches to solving this problem.

The latest consists of task file which is called in a loop per customer. The code for the task file is as follows:

I really don’t like the code as there are way too many loops, and it doesn’t strike me as being very fast/efficient. Which makes me think there must be a better way.

---

- name: "Find the status for the first service IP address {{ health_monitor_test.ServiceIP1 }}."
  no_log: true
  ansible.builtin.set_fact:
    health_check_service1_status: "{{ checking_service1.status }}"
  loop:
    "{{ health_monitor_status_list }}"
  loop_control:
    loop_var: checking_service1
  when: checking_service1['ip-address'] == health_monitor_test.ServiceIP1

- name: "Find the status for the second service IP address {{ health_monitor_test.ServiceIP2 }}."
  no_log: true
  ansible.builtin.set_fact:
    health_check_service2_status: "{{ checking_service2.status }}"
  loop:
    "{{ health_monitor_status_list }}"
  loop_control:
    loop_var: checking_service2
  when: checking_service2['ip-address'] == health_monitor_test.ServiceIP2

- name: "Find the status for the first web site IP address {{ health_monitor_test.WebServiceIP1 }}."
  no_log: true
  ansible.builtin.set_fact:
    health_check_site1_status: "{{ checking_site1.status }}"
  loop:
    "{{ health_monitor_status_list }}"
  loop_control:
    loop_var: checking_site1
  when: checking_site1['ip-address'] == health_monitor_test.WebServiceIP1

- name: "Find the status for the second web site IP address {{ health_monitor_test.WebServiceIP2 }}."
  no_log: true
  ansible.builtin.set_fact:
    health_check_site2_status: "{{ checking_site2.status }}"
  loop:
    "{{ health_monitor_status_list }}"
  loop_control:
    loop_var: checking_site2
  when: checking_site2['ip-address'] == health_monitor_test.WebServiceIP2

- name: "Create the basic structure that will store the health monitor statuses."
  ansible.builtin.set_fact:
    temp: "{
      'customer' : {{ health_monitor_test.CustomerName }},
      'serviceIp1' : {{ health_check_service1_status | default('undefined') }},
      'service2Ip' : {{ health_check_service2_status | default('undefined') }},
      'site1Ip' : {{ health_check_site1_status | default('undefined') }},
      'site2Ip' : {{ health_check_site2_status | default('undefined') }}
    }"

- name: "Add the heartbeat statuses for customer {{ health_monitor_test.TMC_Manifest_CustomerName }} to the main structure."
  no_log: true
  ansible.builtin.set_fact:
    health_monitor_statuses: "{{ health_monitor_statuses + [temp] }}"