Extract and manipulate dict data to check certificates - Jquery

jeanlabab
December 19, 2022
45 views
0 votes
2 Answers

I struggle on a regular basis with data manipulation in Ansible. I’m not very familiar with Python and dict objects. I found an example that sums up a lot of my misunderstandings.

I would like to verify a list of certificates. In found an example for a single domain in the documentation, I’m just trying to loop over several domain names.

Certs are stored in a folder:

certs/
├── domain.com
│   ├── domain.com.pem
│   └── domain.com.key
└── domain.org
    ├── domain.org.key
    └── domain.org.pem

My playbook is as follow:

---
- name: "check certs"
  hosts: localhost
  gather_facts: no
  vars:
    domain_names:
      - domain.com
      - domain.org
    certs_folder: certs
  tasks: 
    - name: Get certificate information
      community.crypto.x509_certificate_info:
        path: "{{ certs_folder }}/{{ item }}/{{ item }}.pem"
        # for valid_at, invalid_at and valid_in
      register: result_certs
      loop: "{{ domain_names }}"
      failed_when: 0

    - name: Get private key information
      community.crypto.openssl_privatekey_info:
        path: "{{ certs_folder }}/{{ item }}/{{ item }}.key"
      register: result_privatekey
      loop: "{{ domain_names }}"
      failed_when: 0

    - name: Check cert and key match << DOES NOT WORK >>>
      assert: 
        that:
          - result_certs[ {{ item }} ].public_key == result_privatekey[ {{ item }} ].public_key
          # - ... other checks ...
          - not result[ {{ item }} ].expired
      loop: "{{ domain_names }}"

So I get two variables result_certs and result_privatekey, each has a element result which is , if I understand correctly, an array of dicts:

"result_certs": {
        "changed": false,
        "msg": "All items completed",
        "results": [
            {
                "expired": false,
                "item": "domain.org",
                "public_key": "<<PUBLIC KEY>>"
            },
            {
                "expired": false,
                "item": "domain.com",
                "public_key": "<<PUBLIC KEY>>"
            }
        ],
        "skipped": false
    }

"result_privatekey": {
    "changed": false,
    "msg": "All items completed",
    "results": [
    {
        "item": "domain.org",
        "public_key": "<< PUBLIC KEY >>"
    },
    {
        "item": "domain.com",
        "public_key": "<< PUBLIC KEY >>"
    }
    ],
    "skipped": false
}

How can I refer to each of the dicts elements like result_privatekey.results[the dict where item ='domain.org'].public_key in the assert task?

I feel like I’m missing something, or a documentation page to make it clear to me. I noticed that I particularly struggle with arrays of dicts, and I run into those objects quite often…

I found those resources useful, but not sufficient to get this job done:

EDIT:
map and selectattr are the filters required to solve this problem, although the documentation (including the official ansible doc) is not that clear to me… This is very useful to get many tutorial examples on those two filters if one is struggling as I do.

Answers

- anonymous_user
- December 19, 2022 at 11:52 am
- 0 votes
0
From what you are showing — but you might have more conditions that needs it — I wouldn’t loop on the domain_names in your assertion task, I would rather loop on result_certs.

From there on, you can select the corresponding private key thanks to the selectattr filter.

So, your assertion would become:
```
- assert: 
    that:
      - >-
          item.public_key == (
            result_privatekey.results 
              | selectattr('item', '==', item.item) 
              | first
          ).public_key
      # - ... other checks ...
      - not item.expired
  loop: "{{ result_certs.results }}"
```
Login or Signup to reply.

Given the simplified data for testing

  result_certs:
    changed: false
    msg: All items completed
    results:
      - expired: false
        item: domain.org
        public_key: <<PUBLIC KEY domain.org>>
      - expired: false
        item: domain.com
        public_key: <<PUBLIC KEY domain.com>>
    skipped: false

  result_privatekey:
    changed: false
    msg: All items completed
    results:
      - item: domain.org
        public_key: <<PUBLIC KEY domain.org>>
      - item: domain.com
        public_key: <<PUBLIC KEY domain.com>>
    skipped: false

Declare the list of the domains

  domains: "{{ result_certs.results|
               map(attribute='item')|list }}"

gives

  domains:
  - domain.org
  - domain.com

Q: "How can I refer to each dictionary element?"

A: select the item(s) and map the attribute

    - debug:
        var: pk
      loop: "{{ domains }}"
      vars:
        pk: "{{ result_privatekey.results|
                selectattr('item', '==', item)|
                map(attribute='public_key')|list }}"

gives

TASK [debug] *********************************************************************************
ok: [localhost] => (item=domain.org) => 
  ansible_loop_var: item
  item: domain.org
  pk:
  - <<PUBLIC KEY domain.org>>
ok: [localhost] => (item=domain.com) => 
  ansible_loop_var: item
  item: domain.com
  pk:
  - <<PUBLIC KEY domain.com>>

In the same way, you can compare the keys in the loop

    - debug:
        msg: "{{ pk1 }} == {{ pk2 }}: {{ pk1 == pk2 }}"
      loop: "{{ domains }}"
      vars:
        pk1: "{{ result_privatekey.results|
                 selectattr('item', '==', item)|
                 map(attribute='public_key')|first }}"
        pk2: "{{ result_certs.results|
                 selectattr('item', '==', item)|
                 map(attribute='public_key')|first }}"

gives

TASK [debug] *********************************************************************************
ok: [localhost] => (item=domain.org) => 
  msg: '<<PUBLIC KEY domain.org>> == <<PUBLIC KEY domain.org>>: True'
ok: [localhost] => (item=domain.com) => 
  msg: '<<PUBLIC KEY domain.com>> == <<PUBLIC KEY domain.com>>: True'

Q: "Extract and manipulate dict data to check certificates."

A: There are many options:

For example, create a unique list of all public keys

  pkeys: "{{ result_certs.results|
             zip(result_privatekey.results)|
             map('map', attribute='public_key')|
             map('unique')|flatten }}"

gives

  pkeys:
  - <<PUBLIC KEY domain.org>>
  - <<PUBLIC KEY domain.com>>

To find the redundant keys compare the lists pkeys and domains. Compare the lengths to briefly find out if there are any

  pkeys|length == domains|length

To find the expired domains declare variables

  expired: "{{ result_certs.results|
               map(attribute='expired')|list }}"
  expired_domains: "{{ result_certs.results|
                       selectattr('expired')|
                       map(attribute='item')|list }}"

give

  expired:
  - false
  - false

  expired_domains: []

Then the assert task should look like

    - assert:
        that:
          - expired is not any
          - pkeys|length == domains|length

Example of a complete playbook for testing

- hosts: localhost

  vars:

    result_certs:
      changed: false
      msg: All items completed
      results:
      - expired: false
        item: domain.org
        public_key: <<PUBLIC KEY domain.org>>
      - expired: false
        item: domain.com
        public_key: <<PUBLIC KEY domain.com>>
      skipped: false

    result_privatekey:
      changed: false
      msg: All items completed
      results:
      - item: domain.org
        public_key: <<PUBLIC KEY domain.org>>
      - item: domain.com
        public_key: <<PUBLIC KEY domain.com>>
      skipped: false

    domains: "{{ result_certs.results|
                 map(attribute='item')|list }}"
    pkeys: "{{ result_certs.results|
               zip(result_privatekey.results)|
               map('map', attribute='public_key')|
               map('unique')|flatten }}"
    expired: "{{ result_certs.results|
                 map(attribute='expired')|list }}"
    expired_domains: "{{ result_certs.results|
                         selectattr('expired')|
                         map(attribute='item')|list }}"

  tasks:

    - debug:
        var: domains

    - debug:
        var: pkeys

    # How can I refer to each of the dicts elements?
    - debug:
        var: pk
      loop: "{{ domains }}"
      vars:
        pk: "{{ result_privatekey.results|
                selectattr('item', '==', item)|
                map(attribute='public_key')|list }}"

    # How can I compare private keys?
    - debug:
        msg: "{{ pk1 }} == {{ pk2 }}: {{ pk1 == pk2 }}"
      loop: "{{ domains }}"
      vars:
        pk1: "{{ result_privatekey.results|
                 selectattr('item', '==', item)|
                 map(attribute='public_key')|first }}"
        pk2: "{{ result_certs.results|
                 selectattr('item', '==', item)|
                 map(attribute='public_key')|first }}"
    - debug:
        var: expired
    - debug:
        var: expired_domains

    - assert:
        that:
          - expired is not any
          - pkeys|length == domains|length

gives

PLAY [localhost] *****************************************************************************

TASK [debug] *********************************************************************************
ok: [localhost] => 
  domains:
  - domain.org
  - domain.com

TASK [debug] *********************************************************************************
ok: [localhost] => 
  pkeys:
  - <<PUBLIC KEY domain.org>>
  - <<PUBLIC KEY domain.com>>

TASK [debug] *********************************************************************************
ok: [localhost] => (item=domain.org) => 
  ansible_loop_var: item
  item: domain.org
  pk:
  - <<PUBLIC KEY domain.org>>
ok: [localhost] => (item=domain.com) => 
  ansible_loop_var: item
  item: domain.com
  pk:
  - <<PUBLIC KEY domain.com>>

TASK [debug] *********************************************************************************
ok: [localhost] => (item=domain.org) => 
  msg: '<<PUBLIC KEY domain.org>> == <<PUBLIC KEY domain.org>>: True'
ok: [localhost] => (item=domain.com) => 
  msg: '<<PUBLIC KEY domain.com>> == <<PUBLIC KEY domain.com>>: True'

TASK [debug] *********************************************************************************
ok: [localhost] => 
  expired:
  - false
  - false

TASK [debug] *********************************************************************************
ok: [localhost] => 
  expired_domains: []

TASK [assert] ********************************************************************************
ok: [localhost] => changed=false 
  msg: All assertions passed

PLAY RECAP ***********************************************************************************
localhost: ok=7    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The next option is to create a structure for this purpose. For example, put into the lists the attributes which might have different values (i.e. public_key in this case). Merge the dictionaries by the domain and append unique attributes. Declare the variables

  l1: "{{ result_certs.results|json_query('[].{item: item,
                                               expired: expired,
                                               pk: [public_key]}') }}"
  l2: "{{ result_privatekey.results|json_query('[].{item: item,
                                                    pk: [public_key]}') }}"
  lm: "{{ [l1, l2]|community.general.lists_mergeby('item', 
                                                    list_merge='append_rp') }}"

gives

  lm:
  - expired: false
    item: domain.com
    pk:
    - <<PUBLIC KEY domain.com>>
  - expired: false
    item: domain.org
    pk:
    - <<PUBLIC KEY domain.org>>

Use this structure to compare any attributes. For example, declare

  exprd: "{{ lm|map(attribute='expired')|list }}"
  pkeys: "{{ lm|map(attribute='pk')|
                map('length')|sum }}"

gives

  exprd:
  - false
  - false

  pkeys: '2'

Then, use it in the conditions

    - assert:
        that:
          - exprd is not any
          - pkeys|int == lm|length

Example of a complete playbook for testing

- hosts: localhost

  vars:

    result_certs:
      changed: false
      msg: All items completed
      results:
      - expired: false
        item: domain.org
        public_key: <<PUBLIC KEY domain.org>>
      - expired: false
        item: domain.com
        public_key: <<PUBLIC KEY domain.com>>
      skipped: false

    result_privatekey:
      changed: false
      msg: All items completed
      results:
      - item: domain.org
        public_key: <<PUBLIC KEY domain.org>>
      - item: domain.com
        public_key: <<PUBLIC KEY domain.com>>
      skipped: false


    l1: "{{ result_certs.results|json_query('[].{item: item,
                                                 expired: expired,
                                                 pk: [public_key]}') }}"
    l2: "{{ result_privatekey.results|json_query('[].{item: item,
                                                      pk: [public_key]}') }}"
    lm: "{{ [l1, l2]|
            community.general.lists_mergeby('item',
                                            list_merge='append_rp') }}"

    exprd: "{{ lm|map(attribute='expired')|list }}"
    pkeys: "{{ lm|map(attribute='pk')|
                  map('length')|sum }}"
  tasks:

    - debug:
        var: l1
    - debug:
        var: l2
    - debug:
        var: lm
    - debug:
        var: exprd
    - debug:
        var: pkeys

    - assert:
        that:
          - exprd is not any
          - pkeys|int == lm|length

In addition to the structure created in option 2) create dictionaries to test the attributes selectively. For example,

  domain_exprd: "{{ lm|items2dict(key_name='item', value_name='expired') }}"
  domain_pkeys: "{{ lm|items2dict(key_name='item', value_name='pk') }}"

gives

  domain_exprd:
    domain.com: false
    domain.org: false

  domain_pkeys:
    domain.com:
    - <<PUBLIC KEY domain.com>>
    domain.org:
    - <<PUBLIC KEY domain.org>>

Please signup or login to give your own answer.

Click here to cancel reply.

Extract and manipulate dict data to check certificates – Jquery

Answers