Showing posts with label Ansible. Show all posts
Showing posts with label Ansible. Show all posts

Wednesday, November 20, 2019

Ansible - dictionaries vs. lists

When working with more complex variables in Ansible (often based on yaml input files) you always have to be aware whether the variable is a list or a dictionary. Applicable filters and methods differ and can lead to errors or unexpected results.

Example input yaml

Here I am defining two variables x and y with some sub elements.
At first glance there doesn't seem to be much difference and if you are about to design a yaml for whatever purpose both solutions might seem interchangeable. The difference lies in its usage which we will see below.
x:
  b1:
    c1: 1
    c2: "aaa"
    c3:
  b2:
    c2: "bbb"
    c3: 5

y:
  - b1:
      c1: 1
      c2: "aaa"
      c3:
  - b2:
      c2: "bbb"
      c3: 5
Call the file dict.yml.

How to check the variable type

Lately there is a new filter in Ansible called type_debug which I find incredibly useful when in doubt.
(unfortunately it was not available in Ansible 1.x, it would have saved me a lot of time)
- hosts: localhost

  tasks:
  - include_vars: dict.yml

  - debug:
      msg: "x: {{x | type_debug}} / y: {{y |type_debug}}"
will show
TASK [debug] ************************************************************************************
ok: [localhost] => {
    "msg": "x: dict / y: list"
}
i.e. I have a dictionary and a list.

How to access the elements

The elements of
  • a dictionary are accessed by name i.e. x['b1']
  • a list are accessed by position i.e. y[0]
    Something like x[0] or y['b1'] would generate a VARIABLE IS UNDEFINED.
    I also show the variable type of the sub elements.
    - hosts: localhost
    
      tasks:
      - include_vars: dict.yml
      
      - debug:
          msg: "{{x['b1'] | type_debug}}"
    
      - debug:
          var: x['b1']
    
      - debug:
          msg: "{{y[0] | type_debug}}"
    
      - debug:
          var: y[0]
    
    will show
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "msg": "dict"
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "x['b1']": {
            "c1": 1,
            "c2": "aaa",
            "c3": null
        }
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "msg": "dict"
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "y[0]": {
            "b1": {
                "c1": 1,
                "c2": "aaa",
                "c3": null
            }
        }
    }
    
    
    You should also note the difference in the result. They are both dictionaries but in the x-case we get a simple dictionary with 3 elements whereas in the y-case we get a dictionary with one element b1 which a sub element of type dictionary.

    How to loop through sub elements

    You can loop easily through the elements by supplying the variable to with_items. The distinction is in what you get as an item. with_item provides
  • dictionary elements as strings and you need to access the sub elements via the {{x[item]}} method
  • list elements are dictionaries
    - hosts: localhost
    
      tasks:
      - include_vars: dict.yml
      
      - debug:
          msg: "{{item}}: {{item|type_debug}} / {{x[item]}}: {{x[item]|type_debug}}"
        with_items: "{{x}}"
    
      - debug:
          msg: "{{item}}: {{item|type_debug}}"
        with_items: "{{y}}"
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => (item=b1) => {
        "msg": "b1: AnsibleUnsafeText / {u'c3': None, u'c2': u'aaa', u'c1': 1}: dict"
    }
    ok: [localhost] => (item=b2) => {
        "msg": "b2: AnsibleUnsafeText / {u'c3': 5, u'c2': u'bbb'}: dict"
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => (item={u'b1': {u'c3': None, u'c2': u'aaa', u'c1': 1}}) => {
        "msg": "{u'b1': {u'c3': None, u'c2': u'aaa', u'c1': 1}}: dict"
    }
    ok: [localhost] => (item={u'b2': {u'c3': 5, u'c2': u'bbb'}}) => {
        "msg": "{u'b2': {u'c3': 5, u'c2': u'bbb'}}: dict"
    }
    

    How to access the bottommost elements

    Say we want to access the value of c2 of b1 for both x and y. There is in both cases the bracket and the dot approach for the dictionary sub elements. In the y-case you need to supply the list position too but it also can be used with the dot approach.
    - hosts: localhost
    
      tasks:
      - include_vars: dict.yml
      
      - debug:
          var: x['b1']['c2']
    
      - debug:
          var: x.b1.c2
    
      - debug:
          var: y[0]['b1']['c2']
    
      - debug:
          var: y.0.b1.c2
    
    will lead to
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "x['b1']['c2']": "aaa"
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "x.b1.c2": "aaa"
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "y[0]['b1']['c2']": "aaa"
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "y.0.b1.c2": "aaa"
    }
    

    Conclusion

    Filters like join, unique, setaddr, map etc. only make sense for the correct variable type. You always need to know what you are dealing with and thus you will be able to create working playbooks faster. It might also influence your design decision when you want to map data into a fitting yaml.

    Why did I write this article

    When I am writing Ansible playbooks they are often based on input yaml files (sometimes my own, sometimes from others) and also often these yaml files contain complex structures which need to be parsed and interpreted correctly.
    I am parsing complex structures by creating intermediate steps and creating new variables with set_fact which contain sub structures of the original one. A common mistake I make is that at certain points in my code I am not sure whether the variable in use is a list or a dictionary. Subsequently when I am using a method or filter this can lead to an error or - worse - to a valid but incorrect result (e.g. an "empty" variable) which will lead to further content errors downstream and the end result is puzzling.
    So I thought for my sake and the sake of the reader a little summary article would help, in particular since I find the Ansible documentation not always as helpful as it could be.
  • Thursday, February 14, 2019

    Ansible tags with 'include'

    When using the include directive things do not quite work as I would expect when I also add a tag to the include section.

    Note: this was tested with Ansible 2.5.2 and python 2.7.15

    Scenario

    I split the playbook of the previous example into two:
    The tasks (tagged in the same fashion as before) have been moved to a new file include_this.yml:
    ---
    
    - name: Debug No Tag
      debug:
        msg: "No tag"
    
    - name: Debug Tag A
      debug:
        msg: "Tag A"
      tags:
        - tagA
    
    - name: Debug Tag B
      debug:
        msg: "Tag B"
      tags:
        - tagB
    
    - name: Debug Tag A and B
      debug:
        msg: "Tag A and B"
      tags:
        - tagA
        - tagB
    
    The playbook playbook.yml has been reduced to two tasks:
    • one debug task
    • one include task
    I am testing two versions: the second one has an additional tag for the include directive.

    Version 1Version 2
    - name: Debug Playbook
      hosts: localhost
      tasks:
        - name: Debug No Tag in Playbook
          debug:
            msg: "No tag in playbook"
        - name: Include
          include: include_this.yml
    
    - name: Debug Playbook
      hosts: localhost
      tasks:
        - name: Debug No Tag in Playbook
          debug:
            msg: "No tag in playbook"
        - name: Include
          include: include_this.yml
          tags:
            - tagA
    

    The playbook is run as

    ansible-playbook [--tags tagA] [--skip-tags tagB] playbook.yml

    Results for version 1

    If called with --tags or --skip-tags version 1 delivers the same results as described in my previous post when the include file was part of the playbook.
    The additional 5th task in playbook.yml is executed in case of --skip-tags tagB or when no arguments are supplied.
    no args
    TASK [Debug No Tag in Playbook] 
    TASK [Debug No Tag] 
    TASK [Debug Tag A] 
    TASK [Debug Tag B]
    TASK [Debug Tag A and B]
    
    --tags tagA
    TASK [Debug Tag A] 
    TASK [Debug Tag A and B]
    
    --tags tagA --skip-tags tagB
    TASK [Debug Tag A] 
    
    --skip-tags tagB
    TASK [Debug No Tag in Playbook]
    TASK [Debug No Tag] 
    TASK [Debug Tag A] 
    
    Now I am reversing tagA and tagB in the call which leads to the expected results.
    --tags tagB
    TASK [Debug Tag B] 
    TASK [Debug Tag A and B]
    
    --tags tagB --skip-tags tagA
    TASK [Debug Tag B] 
    

    Results for version 2

    no args
    TASK [Debug No Tag in Playbook] 
    TASK [Debug No Tag] 
    TASK [Debug Tag A] 
    TASK [Debug Tag B]
    TASK [Debug Tag A and B]
    
    Same as version 1.
    All five tasks are being executed for both version of the playbook. This works as expected. No restrictions of any sort apply.
    --tags tagA
    TASK [Debug No Tag] 
    TASK [Debug Tag A] 
    TASK [Debug Tag B]
    TASK [Debug Tag A and B]
    
    All four task of the included files are being executed. No filtering by tagA for the included file takes place. This was a complete suprise to me. It seems that the tagging of the include step supersedes somehow the tags of the included file.
    --tags tagA --skip-tags tagB
    TASK [Debug No Tag]
    TASK [Debug Tag A] 
    
    The negative filter is applied to the result before and leaves two tasks for execution.
    --skip-tags tagB
    TASK [Debug No Tag in Playbook]
    TASK [Debug No Tag] 
    TASK [Debug Tag A] 
    
    Same as version 1.
    The two tasks with tagB are skipped and the other three are executed.
    --tags tagB
    TASK [Debug Tag B]
    TASK [Debug Tag A and B]
    
    Same as version 1.
    The two tasks with tagB are executed and the tag setting in the include directive is not taken into account.
    --tags tagB --skip-tags tagA
    ... nothing ...
    Now here is a surprise. The --skip-tags tagA has skipped the whole include file and no task is being executed at all.

    An explanation using sets

    Let's look at it from a set perspective.
    --tags tagA--skip-tags tagA
    These two results are complementary and - if joined - build the complete set.
    TASK [Debug No Tag] 
    TASK [Debug Tag A] 
    TASK [Debug Tag B]
    TASK [Debug Tag A and B]
    
    TASK [Debug No Tag in Playbook] 
    Here the same complimentary sets for tagB.
    --tags tagB--skip-tags tagB
    TASK [Debug Tag B]
    TASK [Debug Tag A and B]
    
    TASK [Debug No Tag in Playbook]
    TASK [Debug No Tag] 
    TASK [Debug Tag A]  
    Now any invocation of --skip-tags leads to an intersection with the respective sets.
    Example: --tags tagB --skip-tags tagA: the sets have no task in common and thus nothing will be executed.

    Conclusion

    My idea was that a tag for the include directive would determine whether the include happens or not. Obviously wrong. It seems that it only determines which other tasks in the playbook.yml are executed i.e. it prohibits the untagged task from being run. The include takes place in any case (see the examples when using tagB either in --tags or --skip-tags) but in a strange way since only other tags than the supplied one are applied. I find this hard to remember or explain so my personal rule of mutual tag exclusion will be:
    • tagged include section => no tags in the included file
    • tags in the included file => no tag for the include section

    Ansible tags

    Lately I got a little confused about properly using Ansible tags so I thought I spend a couple of minutes to create a simple example to show how to use them and also explain the pitfalls (at least how I see them).

    Note: this was tested with Ansible 2.5.2 and python 2.7.15

    Scenario

    My example scenario is simple:
    • I have a small playbook with 4 tasks: one task without tag, two tasks with a different tag each and the 4th task with both tags
      - name: Debug Playbook
        hosts: localhost
        tasks:
          - name: Debug No Tag
            debug:
              msg: "No tag"
      
          - name: Debug Tag A
            debug:
              msg: "Tag A"
            tags:
              - tagA
      
          - name: Debug Tag B
            debug:
              msg: "Tag B"
            tags:
              - tagB
      
          - name: Debug Tag A and B
            debug:
              msg: "Tag A and B"
            tags:
              - tagA
              - tagB
      
    • I am running 4 invocations of the playbook with a mix of --tags and --skip-tags to show the effects on the possible outcomes

    Results

    • No tag arguments:
      ansible-playbook playbook.yml
      All tasks are executed.
      TASK [Debug No Tag] *****************************
      ok: [localhost] => {
          "msg": "No tag"
      }
      
      TASK [Debug Tag A] ******************************
      ok: [localhost] => {
          "msg": "Tag A"
      }
      
      TASK [Debug Tag B] ******************************
      ok: [localhost] => {
          "msg": "Tag B"
      }
      
      TASK [Debug Tag A and B] ************************
      ok: [localhost] => {
          "msg": "Tag A and B"
      }
      
    • Supply --tags tagA:
      ansible-playbook --tags tagA playbook.yml
      All tasks are executed where tagA is listed. This works as expected: --tags acts as a filter.
      TASK [Debug Tag A] ******************************
      ok: [localhost] => {
          "msg": "Tag A"
      }
      
      TASK [Debug Tag A and B] ************************
      ok: [localhost] => {
          "msg": "Tag A and B"
      }
      
    • Use both --tags and --skip-tags:
      ansible-playbook --tags tagA --skip-tags tagB playbook.yml
      Only the task tagged tagA is executed. The task with both tags is skipped.
      This actually got me confused. Now when writing this blog it seems obvious: --skip-tags acts as a negative filter i.e. it will skip all tasks where tagB is mentioned.
      TASK [Debug Tag A] ******************************
      ok: [localhost] => {
          "msg": "Tag A"
      }
      
    • Supply --skip-tags tagB:
      ansible-playbook --skip-tags tagB playbook.yml
      All tasks are skipped where tagB is listed, in particular the task without tags is also executed.
      TASK [Debug No Tag] *****************************
      ok: [localhost] => {
          "msg": "No tag"
      }
      
      TASK [Debug Tag A] ******************************
      ok: [localhost] => {
          "msg": "Tag A"
      }
      

    To remember

    • As soon as you switch to using tags via --tags none of the untagged tasks will be executed any longer. If you want the untagged task to be executed in any case you need to supply all possible tags of your playbooks.
    • When mixing --tags and --skip-tags in one call both filters work in conjunction: --tags selects all tasks with the given tags(s), --skip-tags reduces this set by all tasks with the given skip tags.

    The scenario can get a bit trickier when invoking include sections. I will explain this in another blog.