Wednesday, November 20, 2019

Ansible - dictionaries vs. lists

When working with more complex variables in Ansible (often based on yaml input files) you always have to be aware whether the variable is a list or a dictionary. Applicable filters and methods differ and can lead to errors or unexpected results.

Example input yaml

Here I am defining two variables x and y with some sub elements.
At first glance there doesn't seem to be much difference and if you are about to design a yaml for whatever purpose both solutions might seem interchangeable. The difference lies in its usage which we will see below.
x:
  b1:
    c1: 1
    c2: "aaa"
    c3:
  b2:
    c2: "bbb"
    c3: 5

y:
  - b1:
      c1: 1
      c2: "aaa"
      c3:
  - b2:
      c2: "bbb"
      c3: 5
Call the file dict.yml.

How to check the variable type

Lately there is a new filter in Ansible called type_debug which I find incredibly useful when in doubt.
(unfortunately it was not available in Ansible 1.x, it would have saved me a lot of time)
- hosts: localhost

  tasks:
  - include_vars: dict.yml

  - debug:
      msg: "x: {{x | type_debug}} / y: {{y |type_debug}}"
will show
TASK [debug] ************************************************************************************
ok: [localhost] => {
    "msg": "x: dict / y: list"
}
i.e. I have a dictionary and a list.

How to access the elements

The elements of
  • a dictionary are accessed by name i.e. x['b1']
  • a list are accessed by position i.e. y[0]
    Something like x[0] or y['b1'] would generate a VARIABLE IS UNDEFINED.
    I also show the variable type of the sub elements.
    - hosts: localhost
    
      tasks:
      - include_vars: dict.yml
      
      - debug:
          msg: "{{x['b1'] | type_debug}}"
    
      - debug:
          var: x['b1']
    
      - debug:
          msg: "{{y[0] | type_debug}}"
    
      - debug:
          var: y[0]
    
    will show
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "msg": "dict"
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "x['b1']": {
            "c1": 1,
            "c2": "aaa",
            "c3": null
        }
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "msg": "dict"
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "y[0]": {
            "b1": {
                "c1": 1,
                "c2": "aaa",
                "c3": null
            }
        }
    }
    
    
    You should also note the difference in the result. They are both dictionaries but in the x-case we get a simple dictionary with 3 elements whereas in the y-case we get a dictionary with one element b1 which a sub element of type dictionary.

    How to loop through sub elements

    You can loop easily through the elements by supplying the variable to with_items. The distinction is in what you get as an item. with_item provides
  • dictionary elements as strings and you need to access the sub elements via the {{x[item]}} method
  • list elements are dictionaries
    - hosts: localhost
    
      tasks:
      - include_vars: dict.yml
      
      - debug:
          msg: "{{item}}: {{item|type_debug}} / {{x[item]}}: {{x[item]|type_debug}}"
        with_items: "{{x}}"
    
      - debug:
          msg: "{{item}}: {{item|type_debug}}"
        with_items: "{{y}}"
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => (item=b1) => {
        "msg": "b1: AnsibleUnsafeText / {u'c3': None, u'c2': u'aaa', u'c1': 1}: dict"
    }
    ok: [localhost] => (item=b2) => {
        "msg": "b2: AnsibleUnsafeText / {u'c3': 5, u'c2': u'bbb'}: dict"
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => (item={u'b1': {u'c3': None, u'c2': u'aaa', u'c1': 1}}) => {
        "msg": "{u'b1': {u'c3': None, u'c2': u'aaa', u'c1': 1}}: dict"
    }
    ok: [localhost] => (item={u'b2': {u'c3': 5, u'c2': u'bbb'}}) => {
        "msg": "{u'b2': {u'c3': 5, u'c2': u'bbb'}}: dict"
    }
    

    How to access the bottommost elements

    Say we want to access the value of c2 of b1 for both x and y. There is in both cases the bracket and the dot approach for the dictionary sub elements. In the y-case you need to supply the list position too but it also can be used with the dot approach.
    - hosts: localhost
    
      tasks:
      - include_vars: dict.yml
      
      - debug:
          var: x['b1']['c2']
    
      - debug:
          var: x.b1.c2
    
      - debug:
          var: y[0]['b1']['c2']
    
      - debug:
          var: y.0.b1.c2
    
    will lead to
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "x['b1']['c2']": "aaa"
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "x.b1.c2": "aaa"
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "y[0]['b1']['c2']": "aaa"
    }
    
    TASK [debug] ************************************************************************************
    ok: [localhost] => {
        "y.0.b1.c2": "aaa"
    }
    

    Conclusion

    Filters like join, unique, setaddr, map etc. only make sense for the correct variable type. You always need to know what you are dealing with and thus you will be able to create working playbooks faster. It might also influence your design decision when you want to map data into a fitting yaml.

    Why did I write this article

    When I am writing Ansible playbooks they are often based on input yaml files (sometimes my own, sometimes from others) and also often these yaml files contain complex structures which need to be parsed and interpreted correctly.
    I am parsing complex structures by creating intermediate steps and creating new variables with set_fact which contain sub structures of the original one. A common mistake I make is that at certain points in my code I am not sure whether the variable in use is a list or a dictionary. Subsequently when I am using a method or filter this can lead to an error or - worse - to a valid but incorrect result (e.g. an "empty" variable) which will lead to further content errors downstream and the end result is puzzling.
    So I thought for my sake and the sake of the reader a little summary article would help, in particular since I find the Ansible documentation not always as helpful as it could be.
  • Tuesday, October 15, 2019

    Identifying and handling Java keystore store types in different Java versions

    Recently I encountered a different behaviour of Java keytool in newer versions of Java (I tested 10.0.2 and 11.0.4) compared to older versions (I tested 7.0_131 and 8.0_144) in regards to identifying store types.

    Generate keystore with various store types

    keytool is able to handle three different store types.
    Basically I am running these commands (regardless which Java version is in place). Store types are not case sensitive.
    keytool -genkey -alias foo -keystore a.keystore -storetype JKS
    keytool -genkey -alias foo -keystore b.keystore -storetype JCEKS
    keytool -genkey -alias foo -keystore c.keystore -storetype PKCS12
    

    In detail:
    I supply a password and simply hit Enter to all the questions being asked except the last one (answer y). I also don't provide a different key password.
    Note that the default store type has also changed between Java versions (JKS in Java 8, PKCS12 in Java 10) so running the above commands without -storetype .. will generate keystores of different type resp. to your Java version in use. It is best to always indicate a store type in order to avoid any disambiguity.

    keytool -genkey -alias foo -keystore a.keystore -storetype jks
    Enter keystore password:
    Re-enter new password:
    What is your first and last name?
    ...
    Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown correct?
      [no]:  y
    
    Enter key password for 
            (RETURN if same as keystore password):
    
    

    Checking the keystore type via keytool -list

    Since keytool -list ... also indicates the store type (before listing the aliases) I am running
    keytool -list -keystore a.keystore
    keytool -list -keystore b.keystore
    keytool -list -keystore c.keystore
    

    Result for Java 7 and 8

    keytool -list -keystore a.keystore
    Enter keystore password:
    
    Keystore type: JKS
    Keystore provider: SUN
    
    Your keystore contains 1 entry
    
    foo, Oct 15, 2019, PrivateKeyEntry,
    Certificate fingerprint (SHA1): 69:B0:E2:F8:D4:C3:77:70:A2:20:AC:BE:51:B1:6A:FF:71:85:AF:4F
    
    keytool -list -keystore b.keystore
    keytool error: java.io.IOException: Invalid keystore format
    
    keytool -list -keystore c.keystore
    keytool error: java.io.IOException: Invalid keystore format
    
    

    i.e. keytool does not recognize the store type in Java 7 and 8
    If you want to see the list of aliases you need to know the store type in advance and indicate it on the command line.

    keytool -list -keystore b.keystore -storetype JCEKS
    Enter keystore password:
    
    Keystore type: JCEKS
    Keystore provider: SunJCE
    
    Your keystore contains 1 entry
    
    foo, Oct 15, 2019, PrivateKeyEntry,
    Certificate fingerprint (SHA1): 87:08:88:32:20:F2:61:11:C3:82:E0:DF:26:76:7B:8A:CD:2C:2C:B8
    
    keytool -list -keystore c.keystore -storetype PKCS12
    Enter keystore password:
    
    Keystore type: PKCS12
    Keystore provider: SunJSSE
    
    Your keystore contains 1 entry
    
    foo, Oct 15, 2019, PrivateKeyEntry,
    Certificate fingerprint (SHA1): E8:C4:B1:F2:E2:C9:7F:9F:D0:10:77:3D:F5:07:30:77:3C:E3:74:8D
    

    Result for Java 10 and 11

    The newer Java versions are able to list any store type.
    keytool -list -keystore a.keystore
    Enter keystore password:
    
    Keystore type: JKS
    Keystore provider: SUN
    
    Your keystore contains 1 entry
    
    foo, Oct 15, 2019, PrivateKeyEntry,
    Certificate fingerprint (SHA-256): 25:EB:EE:65:AC:AD:F5:44:58:90:82:E3:BE:9D:A2:AD:D0:EC:91:50:CB:72:75:C9:03:12:0F:58:4F:A5:FA:44
    
    Warning:
    The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore x.keystore -destkeystore x.keystore -deststoretype pkcs12".
    
    keytool -list -keystore b.keystore
    Enter keystore password:
    
    Keystore type: JCEKS
    Keystore provider: SunJCE
    
    Your keystore contains 1 entry
    
    foo, Oct 15, 2019, PrivateKeyEntry,
    Certificate fingerprint (SHA-256): B2:61:07:4D:D5:12:58:D7:F8:49:BB:1A:7E:69:FC:F2:C4:3E:3A:9B:F3:B0:E0:F0:F5:3E:9F:46:C4:8B:A7:86
    
    Warning:
    The JCEKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore y.keystore -destkeystore y.keystore -deststoretype pkcs12".
    
    keytool -list -keystore c.keystore
    Enter keystore password:
    Keystore type: PKCS12
    Keystore provider: SUN
    
    Your keystore contains 1 entry
    
    foo, Oct 15, 2019, PrivateKeyEntry,
    Certificate fingerprint (SHA-256): B4:E3:38:34:21:27:AC:0F:43:09:46:BC:FE:41:DF:1E:F5:5F:3A:75:75:52:D3:7C:42:A4:F7:57:B5:FC:34:9E
    

    Note that the fingerprints are different and the store types JKS and JCEKS generate a warning.
    The keystore provider for PKCS12 is SunJSSE in Java 7 and SUN in Java 10.


    On UNIX systems you can use the file command to get an indication of the resp. store type.
    file *.keystore
    
    a.keystore: Java KeyStore
    b.keystore: Java JCE KeyStore
    c.keystore: data
    
    i.e. the file "magic" can identify the old JKS and JCEKS keystores but the newest recommended PKCS12 is only shown as data.

    Thursday, September 19, 2019

    Assignment operator in python vs. copy

    Lately I stumbled across the following issue when assigning a variable to another (which was a list) and a change to the original reflected in the second.
    (This is in python 3.7. I am not using the python prompt >>> in the examples below.)

    x = ['a']
    y = x
    x.append('b')
    print(x)
       ['a', 'b']
    print(y)
       ['a', 'b']
    

    An example with int shows this

    x = 5
    y = x
    x = x + 7
    print(x)
      12
    print(y)
      5
    

    What might seem surprising at first glance totally makes sense:
    an assignment '=' is not a copy operation.

    The assignment

    x = something
    
    rather gives the name tag x to the object something.
    y = x
    
    gives the name tag y to the same object which is tagged x.
    There is no copy aka. creation of a new object happening here.

    The conclusion is:
    if the object tagged x is a changeable object any change will be seen in y too.
    So if the object is a list, tuple, set, dict etc. a change to the object (like the append operation) is both visible when either accessing x or y.

    If the object is an integer or a string then it cannot be changed. Operations like addition create a new object.
    x = x + 7 does not change the former object 5 but creates a new object 12.
    y is still pointing to the old object and thus shows the old value.

    If the intention of the original code was to create a copy of a list then a true copy operation should be used instead.

    y = x[:]
    

    Wednesday, March 6, 2019

    How to determine the number of elements of each directory in a directory tree

    Imagine that you have a directory tree with multiple sub directories and files and you want to determine the number of files and directories for each directory in this hierarchy i.e. not just direct elements but all elements of sub directories as well. Here I present a solution just using the UNIX tools find and awk.

    Creating an example directory tree

    With these commands I am creating a simple directory tree with some directories and files which I will use further on.
    mkdir -p somedir/d1/b somedir/d1/c somedir/d2 somedir/d3;    # Create some directories
    touch somedir/d1/b/file1 somedir/d3/file2;                   # Create some files
    ln -s somedir/d3/file2 somedir/d3/z;                             # Create symbolic links
    (cd somedir; ln -s d1 d4_link ;)
    
    find somedir
    somedir
    somedir/d1
    somedir/d1/b
    somedir/d1/b/file1
    somedir/d1/c
    somedir/d2
    somedir/d3
    somedir/d3/file2
    somedir/d3/z
    somedir/d4_link
    
    I want some code to show me that somedir contains 9 elements and somedir/d1 contains 3 elements aso.

    List the complete directory tree with types of files

    On Linux systems with GNU find (unfortunately this does not work on my Mac) one can easily retrieve a list of elements and their file types with the printf formatting options. My example contains six directories (including the topmost somedir), two files and two symbolic links (one to a file and one to a directory).
    find somedir -printf '%y %p\n'
    
    d somedir
    d somedir/d1
    d somedir/d1/b
    f somedir/d1/b/file1
    d somedir/d1/c
    d somedir/d2
    d somedir/d3
    f somedir/d3/file2
    l somedir/d3/z
    l somedir/d4_link
    

    Calculating the number of entries per directory

    First of all the awk command consumes a slightly modified output of the find from above:
    I am adding a slash so that I can use the slash as an awk field delimiter.
    find somedir -printf '%y/%p\n'
    
    ...
    d/somedir/d1/b
    ...
    
    The awk command:
    awk -F/ '
    { 
      x = $2; 
      count[x]++; 
      for(i=3;i<=NF;i++) { x = x FS $i; count[x]++ } 
      type[x] = $1;
    } 
    END {
      for(i in count)  printf "%s %2d %s\n", type[i],count[i]-1,i
    }'
    
    What happens in each step: the first field is the type of the element ( d for directory, f for field etc.), the count array is increased for each occurance of a path.
    d/somedir
    
      x = "somedir"
      count["somedir"] = 1
      # the for loop is not executed for "d/somedir" since NF=2
      type["somedir"} = "d"
    
    d/somedir/d1
      x = "somedir"
      count["somedir"] = 2      # increase count by 1
     
      # NF = 3. The for loop is executed once.
      x = x FS "a" = "somedir/d1"
      count["somedir/d1"] = 1    # first time: 1
    
      type["somedir/d1"] = "d"
    
    d/somedir/d1/b
      x = "somedir"
      count["somedir"] = 3      # increase count by 1
     
      # NF = 4. The for loop is executed twice.
      x = x FS "a" = "somedir/d1"
      count["somedir/d1"] = 2    # increase count by 1
    
      x = x FS "b" = "somedir/d1/b"
      count["somedir/d1/b"] = 1  # first time: 1
    
      type["somedir/d1/b"] = "d"
    

    Here is the combined command sequence also appended by a sort statement for better readability

    find somedir -printf '%y/%p\n'  | 
    awk -F/ '{ x=$2; count[x]++; for(i=3;i<=NF;i++) { x=x FS $i; count[x]++ } type[x]=$1;} 
    END {for(i in count)  printf "%s %2d %s\n", type[i],count[i]-1,i } '|
    sort  -k3,3
    
    d  9 somedir
    d  3 somedir/d1
    d  1 somedir/d1/b
    f  0 somedir/d1/b/file1
    d  0 somedir/d1/c
    d  0 somedir/d2
    d  2 somedir/d3
    f  0 somedir/d3/file2
    l  0 somedir/d3/z
    l  0 somedir/d4_link
    
    So somedir contains 9 elements altogether: 4 direct elements d1, d2, d3 and d4_link and also elements of elements.
    Note that the END statement prints the count minus one since the count was set to one when the directory appeared originally but we want to show only the number of elements i.e. I need to exclude the directory itself.
    Note also that somedir/d4_link (the symbolic link to directory somedir/d1) is not followed and listed as having zero elements. If you want to follow symbolic links to directories with find somedir -follow the calculations will be misleading since - in this example - elements of d1 and d4_link would be calculated twice.

    The counts for non-directory file types should always be zero, they could probably be excluded completely from the output.

    find somedir -printf '%y/%p\n'  | 
    awk -F/ '{ x=$2; count[x]++; for(i=3;i<=NF;i++) { x=x FS $i; count[x]++ } type[x]=$1;} 
    END {for(i in count) if( type[i]=="d") printf "%2d %s\n", count[i]-1,i } '| sort  -k2,2
    
     9 somedir
     3 somedir/d1
     1 somedir/d1/b
     0 somedir/d1/c
     0 somedir/d2
     2 somedir/d3
    

    Usages

    Empty directories

    Add a grep '^d 0' (or adjust the awk code with if clause count[i]==1 etc.)
    find somedir -printf '%y/%p\n'  | 
    awk -F/ '{ x=$2; count[x]++; for(i=3;i<=NF;i++) { x=x FS $i; count[x]++ } type[x]=$1;} 
    END {for(i in count)  printf "%s %2d %s\n", type[i],count[i]-1,i } '|
    grep '^d  0'
    
    d  0 somedir/d1/c
    d  0 somedir/d2
    

    Non-empty directories

    find somedir -printf '%y/%p\n'  | 
    awk -F/ '{ x=$2; count[x]++; for(i=3;i<=NF;i++) { x=x FS $i; count[x]++ } type[x]=$1;} 
    END {for(i in count)  printf "%s %2d %s\n", type[i],count[i]-1,i } '|
    grep '^d .[^0]' | sort -k 3,3
    
    d  9 somedir
    d  3 somedir/d1
    d  1 somedir/d1/b
    d  2 somedir/d3
    

    Wednesday, February 20, 2019

    Using the 'timeout' command

    In Linux there is timeout command (usually in /usr/bin/timeout) which is very handy when you really want to restrict possibly long running commands. Here I want to show a couple of invocations to explain its usage.

    First of all: timeout exits with error code 124 if the timeout has been reached.
    Of particular interest is the handling of exit codes if commands are failing.

    I am playing with various invocations of a timeout setting of 3 seconds and a sleep command of 5 seconds and vice versa.

    Output Exit code
    The timeout threshold is reached.
    timeout 3 sleep 5
    124
    timeout 3 sh -c "sleep 5; exit 1"
    124
    timeout 3 sh -c "echo something; sleep 5; exit 1"
    something124Failure
    but part of the commands ran successfully creating output.
    timeout 3 sh -c " sleep 5; echo something; exit 1"
    124
    The timeout threshold is not reached.
    timeout 5 sleep 3
    0Success.
    The command finished before the timeout threshold.
    timeout 3 sh -c "exit 1"
    1Failure.
    The command finished before the timeout threshold but with an error.
    timeout 3 sh -c "echo something; exit 1"
    something1Failure.
    The commands ran and finished before the timeout threshold but with an error.

    Here is an example that shows how a loop gets executed partially.

    timeout 3 sh -c "for i in 1 2 3; do sleep 1; echo i=\$i;  done"
    
    i=1
    i=2
    
    and exit code 124
    
    In this case one needs to carefully continue and understand possible consequences of a partially executed command chain and how to recover from timeout errors.

    Error handling could be done this way:

    case $? in
    0)
      # all ok
      ;;
    124)
      # this was a timeout
      ;;
    *)
      # some other error induced by the executed command(s)
      ;;
    esac
    

    Maybe this usage is also helpful sometimes: capture only the timeout exit code.

    timeout 3 sh -c "echo something; sleep 5" || { [ $? -eq 124 ] && echo TIMEOUT; }
    echo $?
    
    will show output of the executed command, the timeout exit code check and an exit code 0
    
    something
    TIMEOUT
    0
    
    Interesting to see what happens when the executed command fails. The exit code of the failed command is still available to subsequent checks.
    timeout 3 sh -c "echo something; exit 1" || { [ $? -eq 124 ] && echo TIMEOUT; }
    echo $?
    
    will show output of the executed command(s) and the exit code of the command chain
    
    something
    1
    

    Thursday, February 14, 2019

    Ansible tags with 'include'

    When using the include directive things do not quite work as I would expect when I also add a tag to the include section.

    Note: this was tested with Ansible 2.5.2 and python 2.7.15

    Scenario

    I split the playbook of the previous example into two:
    The tasks (tagged in the same fashion as before) have been moved to a new file include_this.yml:
    ---
    
    - name: Debug No Tag
      debug:
        msg: "No tag"
    
    - name: Debug Tag A
      debug:
        msg: "Tag A"
      tags:
        - tagA
    
    - name: Debug Tag B
      debug:
        msg: "Tag B"
      tags:
        - tagB
    
    - name: Debug Tag A and B
      debug:
        msg: "Tag A and B"
      tags:
        - tagA
        - tagB
    
    The playbook playbook.yml has been reduced to two tasks:
    • one debug task
    • one include task
    I am testing two versions: the second one has an additional tag for the include directive.

    Version 1Version 2
    - name: Debug Playbook
      hosts: localhost
      tasks:
        - name: Debug No Tag in Playbook
          debug:
            msg: "No tag in playbook"
        - name: Include
          include: include_this.yml
    
    - name: Debug Playbook
      hosts: localhost
      tasks:
        - name: Debug No Tag in Playbook
          debug:
            msg: "No tag in playbook"
        - name: Include
          include: include_this.yml
          tags:
            - tagA
    

    The playbook is run as

    ansible-playbook [--tags tagA] [--skip-tags tagB] playbook.yml

    Results for version 1

    If called with --tags or --skip-tags version 1 delivers the same results as described in my previous post when the include file was part of the playbook.
    The additional 5th task in playbook.yml is executed in case of --skip-tags tagB or when no arguments are supplied.
    no args
    TASK [Debug No Tag in Playbook] 
    TASK [Debug No Tag] 
    TASK [Debug Tag A] 
    TASK [Debug Tag B]
    TASK [Debug Tag A and B]
    
    --tags tagA
    TASK [Debug Tag A] 
    TASK [Debug Tag A and B]
    
    --tags tagA --skip-tags tagB
    TASK [Debug Tag A] 
    
    --skip-tags tagB
    TASK [Debug No Tag in Playbook]
    TASK [Debug No Tag] 
    TASK [Debug Tag A] 
    
    Now I am reversing tagA and tagB in the call which leads to the expected results.
    --tags tagB
    TASK [Debug Tag B] 
    TASK [Debug Tag A and B]
    
    --tags tagB --skip-tags tagA
    TASK [Debug Tag B] 
    

    Results for version 2

    no args
    TASK [Debug No Tag in Playbook] 
    TASK [Debug No Tag] 
    TASK [Debug Tag A] 
    TASK [Debug Tag B]
    TASK [Debug Tag A and B]
    
    Same as version 1.
    All five tasks are being executed for both version of the playbook. This works as expected. No restrictions of any sort apply.
    --tags tagA
    TASK [Debug No Tag] 
    TASK [Debug Tag A] 
    TASK [Debug Tag B]
    TASK [Debug Tag A and B]
    
    All four task of the included files are being executed. No filtering by tagA for the included file takes place. This was a complete suprise to me. It seems that the tagging of the include step supersedes somehow the tags of the included file.
    --tags tagA --skip-tags tagB
    TASK [Debug No Tag]
    TASK [Debug Tag A] 
    
    The negative filter is applied to the result before and leaves two tasks for execution.
    --skip-tags tagB
    TASK [Debug No Tag in Playbook]
    TASK [Debug No Tag] 
    TASK [Debug Tag A] 
    
    Same as version 1.
    The two tasks with tagB are skipped and the other three are executed.
    --tags tagB
    TASK [Debug Tag B]
    TASK [Debug Tag A and B]
    
    Same as version 1.
    The two tasks with tagB are executed and the tag setting in the include directive is not taken into account.
    --tags tagB --skip-tags tagA
    ... nothing ...
    Now here is a surprise. The --skip-tags tagA has skipped the whole include file and no task is being executed at all.

    An explanation using sets

    Let's look at it from a set perspective.
    --tags tagA--skip-tags tagA
    These two results are complementary and - if joined - build the complete set.
    TASK [Debug No Tag] 
    TASK [Debug Tag A] 
    TASK [Debug Tag B]
    TASK [Debug Tag A and B]
    
    TASK [Debug No Tag in Playbook] 
    Here the same complimentary sets for tagB.
    --tags tagB--skip-tags tagB
    TASK [Debug Tag B]
    TASK [Debug Tag A and B]
    
    TASK [Debug No Tag in Playbook]
    TASK [Debug No Tag] 
    TASK [Debug Tag A]  
    Now any invocation of --skip-tags leads to an intersection with the respective sets.
    Example: --tags tagB --skip-tags tagA: the sets have no task in common and thus nothing will be executed.

    Conclusion

    My idea was that a tag for the include directive would determine whether the include happens or not. Obviously wrong. It seems that it only determines which other tasks in the playbook.yml are executed i.e. it prohibits the untagged task from being run. The include takes place in any case (see the examples when using tagB either in --tags or --skip-tags) but in a strange way since only other tags than the supplied one are applied. I find this hard to remember or explain so my personal rule of mutual tag exclusion will be:
    • tagged include section => no tags in the included file
    • tags in the included file => no tag for the include section

    Ansible tags

    Lately I got a little confused about properly using Ansible tags so I thought I spend a couple of minutes to create a simple example to show how to use them and also explain the pitfalls (at least how I see them).

    Note: this was tested with Ansible 2.5.2 and python 2.7.15

    Scenario

    My example scenario is simple:
    • I have a small playbook with 4 tasks: one task without tag, two tasks with a different tag each and the 4th task with both tags
      - name: Debug Playbook
        hosts: localhost
        tasks:
          - name: Debug No Tag
            debug:
              msg: "No tag"
      
          - name: Debug Tag A
            debug:
              msg: "Tag A"
            tags:
              - tagA
      
          - name: Debug Tag B
            debug:
              msg: "Tag B"
            tags:
              - tagB
      
          - name: Debug Tag A and B
            debug:
              msg: "Tag A and B"
            tags:
              - tagA
              - tagB
      
    • I am running 4 invocations of the playbook with a mix of --tags and --skip-tags to show the effects on the possible outcomes

    Results

    • No tag arguments:
      ansible-playbook playbook.yml
      All tasks are executed.
      TASK [Debug No Tag] *****************************
      ok: [localhost] => {
          "msg": "No tag"
      }
      
      TASK [Debug Tag A] ******************************
      ok: [localhost] => {
          "msg": "Tag A"
      }
      
      TASK [Debug Tag B] ******************************
      ok: [localhost] => {
          "msg": "Tag B"
      }
      
      TASK [Debug Tag A and B] ************************
      ok: [localhost] => {
          "msg": "Tag A and B"
      }
      
    • Supply --tags tagA:
      ansible-playbook --tags tagA playbook.yml
      All tasks are executed where tagA is listed. This works as expected: --tags acts as a filter.
      TASK [Debug Tag A] ******************************
      ok: [localhost] => {
          "msg": "Tag A"
      }
      
      TASK [Debug Tag A and B] ************************
      ok: [localhost] => {
          "msg": "Tag A and B"
      }
      
    • Use both --tags and --skip-tags:
      ansible-playbook --tags tagA --skip-tags tagB playbook.yml
      Only the task tagged tagA is executed. The task with both tags is skipped.
      This actually got me confused. Now when writing this blog it seems obvious: --skip-tags acts as a negative filter i.e. it will skip all tasks where tagB is mentioned.
      TASK [Debug Tag A] ******************************
      ok: [localhost] => {
          "msg": "Tag A"
      }
      
    • Supply --skip-tags tagB:
      ansible-playbook --skip-tags tagB playbook.yml
      All tasks are skipped where tagB is listed, in particular the task without tags is also executed.
      TASK [Debug No Tag] *****************************
      ok: [localhost] => {
          "msg": "No tag"
      }
      
      TASK [Debug Tag A] ******************************
      ok: [localhost] => {
          "msg": "Tag A"
      }
      

    To remember

    • As soon as you switch to using tags via --tags none of the untagged tasks will be executed any longer. If you want the untagged task to be executed in any case you need to supply all possible tags of your playbooks.
    • When mixing --tags and --skip-tags in one call both filters work in conjunction: --tags selects all tasks with the given tags(s), --skip-tags reduces this set by all tasks with the given skip tags.

    The scenario can get a bit trickier when invoking include sections. I will explain this in another blog.

    A list of Google fonts

    Here is a list of Google fonts. It is constructed with Javascript and CSS and dynamically embedded into this blog.

    Some fonts can be made more effective in bigger sizes or by applying font effects but that would be too much for this overview.
    Enjoy and find your font.

    Monday, February 11, 2019

    How to use Google fonts (even in this blog)

    Google provides a huge list of fonts and it describes how to use them in detail, in particular applying font effects and more.

    I thought it worthwhile to capture the essence of using fonts and also I wanted to see if I could use Google fonts in this blog.

    Using Google fonts boils down to three things actually:

    • load the font from Google by supplying a link in the <head> section of your HTML document
      <link href="https://fonts.googleapis.com/css?family=Amarante" rel="stylesheet">
      
    • defining a corresponding style e.g. a class style with a name
      <style>
        .coolFont { font-family: Amarante }
      </style>
      
    • applying this font to any element with
      < ... class="coolFont" ... >
      

    Here is a Javascript code snippet (which I actually included in the HTML text) which dynamically loads three fonts and defines three correponding styles. It applies each of the styles to an h1 element.

    <script>
    var fonts = [ 'Acme', 'Amarante', 'Audiowide' ]
    
    for( i=0; i<fonts.length; i++ ) {
      var style = document.createElement( 'STYLE' );
      // Define class style '.font0', '.font1' etc.
      style.innerHTML = ".font" + i + "{ font-family: " + fonts[i] + ";}"
      document.getElementsByTagName('head')[0].appendChild( style );
    
      var  link = document.createElement( 'LINK' );
      link.setAttribute( 'href', 'https://fonts.googleapis.com/css?family=' + encodeURI( fonts[i] ) );
      link.setAttribute( 'rel', "stylesheet" ) ;
      document.getElementsByTagName('head')[0].appendChild( link );
    }
    </script>
    
    <h1 class="font0">Cool font, dude! </h1>
    <h1 class="font1">Cool font, dude! </h1>
    <h1 class="font2">Cool font, dude! </h1>
    
    This will create a links like this
    <link href="https://fonts.googleapis.com/css?family=Amarante" rel="stylesheet">
    
    and styles
    .font1 {
      font-family: Amarante;
    }
    

    and here is the result

    Cool font, dude!

    Cool font, dude!

    Cool font, dude!

    Thursday, January 24, 2019

    How to get coloured output from 'find'

    When I need to get coloured output from shell scripts (e.g. in error messages) I am using the oldstyle escape sequences.

    Example:

    printf "\e[31mERROR - some error description\e[0m\n"
    
    which will result in
    ERROR - some error description
    
    \e[31m is basically ESCAPE and a colour code (31 for red, see ANSI escape codes) for colour terminals. \e[0m ends the colouring.

    I wanted to use the colour escape sequences in the printf of find.

    This does not work when using \e and an error is thrown:

    find . -printf "\e[31mSome Output\n" -quit
    find: warning: unrecognized escape `\e'
    \e[31mSome Output
    
    The solution is to replace \e by its octal representation \033 (see e.g. ASCII table).
    find . -printf "\033[31mSome Output\n" -quit
    Some Output
    

    Of course the find command in the example is not really useful (it quits right away) and it is only used to showcase the issue. Here is a more useful invocation:

    find empty and non-empty sub directories whereas non-empty should be coloured.

    # Create example directories
    mkdir -p tmp/a tmp/b tmp/c; touch tmp/b/foo
    
    # Find
    find tmp -type d \( -empty -printf "%-10p empty\n" \) -o \
    \( ! -empty -printf "\033[31m%-10p not empty\033[0m\n" \)
    tmp        not empty
    tmp/a      empty
    tmp/b      not empty
    tmp/c      empty
    
    i.e. I can distuingish in find between different cases and apply different colours according to my needs.