tag:blogger.com,1999:blog-71943841725753291092024-03-06T01:01:46.258+01:00Andreas' Technical TidbitsA collection of mostly software related bits and pieces out of my world (UNIX, scripting, OpenOffice.org, etc.)Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.comBlogger86125tag:blogger.com,1999:blog-7194384172575329109.post-37367044083359578172020-07-08T15:29:00.000+02:002020-07-09T17:58:53.074+02:00Ansible - convert yaml file to jsonUsing the Ansible filters <B>from_yaml</B> and <B>to_json</B> it is easy to construct a task to <B>convert a yaml file to JSON</B>.
<P>
Here is the Ansible playbook <code>yaml_to_json.yml</code> (in a real world example the names of the files would probably be parameterized).
<PRE>
---
- hosts: localhost
tasks:
- shell: cat my.yaml
register: result
- set_fact:
myvar: "{{ result.stdout | from_yaml | to_json }}"
- copy:
content: "{{ myvar }}"
dest: my.json
</PRE>
<P>
Here is an example of a yaml input file which contains some special characters, lists, strings, numbers etc.
<PRE>
---
xx:
- a:
- b:
A: aadf7{fdfd"öög
B: [ 'asdfadsf*5%@@@"masdf' , 123456 ]
yy:
n:
C: 'AAdf7{fdfd"öög'
D: [ 'ASdfadsf*5%@@@"masdf' , 789.56 ]
m:
</PRE>
<P>
When run via <code>ansible-playbook yaml_to_json.yml</code> the
JSON output file below (run via <code>jq .</code>) is generated.
<PRE>
{
"yy": {
"m": null,
"n": {
"C": "AAdf7{fdfd\"\\u00f6\\u00f6g",
"D": [
"ASdfadsf*5%@@@\"masdf",
789.56
]
}
},
"xx": [
{
"a": null
},
{
"b": {
"A": "aadf7{fdfd\"\\u00f6\\u00f6g",
"B": [
"asdfadsf*5%@@@\"masdf",
123456
]
}
}
]
}
</PRE>
Note how lists and dictionaries and special characters are put into the resp. JSON format.
<P>
Also note that I am using the <code>Ansible copy</code> module.
A <code>shell: echo "{{ myvar }}" > my.json</code> does <i>not</i> work since it does not take care of the correct quote and special characters subsitutions.Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com2tag:blogger.com,1999:blog-7194384172575329109.post-41577558865321156112020-03-20T21:13:00.001+01:002020-03-20T21:13:54.433+01:00Different 'exit' behaviour between bash and Korn shell in command pipelinesWhen running command pipelines it is sometimes convenient to break out of a sub process and <B>exit</B> a shell script.
<P>
There are differences though between different types of <B>shell</B> and I want to show this behaviour and its consequences , in particular how common expectations might be met or not, and also suggest solutions.
<P>
Here is my example. It is a simple script: a command pipeline consisting of a <B>printf</B> (printing two lines) followed by a <B>while</B>, wrapped by a starting and closing <B>echo</B>
<PRE>
echo Before pipeline
printf "text1\ntext2\n" | while read line ; do
# Do something useful with 'line'
echo $line
done
echo After pipeline
</PRE>
and the output - with whichever shell - is
<PRE>
Before pipeline
text1
text2
After pipeline
</PRE>
<P>
Now let's modify this script and add an <B>exit</B> into the <B>while</B> loop so that it looks like this
<PRE>
echo Before pipeline
printf "text1\ntext2\n" | while read line ; do
# Do something useful with 'line'
echo $line
exit 1
done
echo After pipeline
</PRE>
As artificial as this example might look you can simply assume that there are more complex things happening in the <B>while</B> loop and under certain conditions one might want to exit the loop.
<P>
Here are the results for <B>bash</B> and Korn shell <B>ksh</B>
<table align="center" border="1" cellpadding="10" cellspacing="0"><tbody>
<tr> <th align="center">bash<BR>#!/bin/bash</th><th align="center">ksh<BR>#!/bin/ksh</th></tr>
<tr> <td valign=top><pre>
Before pipeline
text1
After pipeline
</pre></td> <td valign=top><pre>
Before pipeline
text1
</pre></td> </tr>
<tr> <td>Exit code: <B>0</B></td> <td>Exit code: <B>1</B></td></tr>
<tr> <td align="center" colspan="2"><B>Common behaviour</B>: both shells leave the loop and neither prints the second line 'text2'</td></tr>
<tr> <td>bash <B>continues</B> with the commands after the while loop<BR>(and exits with the result of the last 'echo' command)</td> <td>ksh <B>exits</B> the whole script</td></tr>
</tbody></table><br />
<P>
While the behaviour of <B>ksh</B> seems more natural (exit means exit everything) the <B>bash</B> behaviour can be explained when considering that the <B>while</B> loop is a sub shell with its own scope as if it would be a separate shell script. Exiting the loop means to return to the <B>parent</B>.
This also means that the <B>parent</B> script can capture this exit code and thus the solution is a check after the loop.
<PRE>
echo Before pipeline
printf "text1\ntext2\n" | while read line ; do
# Do something useful with 'line'
echo $line
exit 1
done
[ $? -ne 0 ] && echo ERROR && exit 2
echo After pipeline
</PRE>
The result for bash: exit code <B>2</B> and the output
<PRE>
Before pipeline
text1
ERROR
</PRE>
The exit code check line is meaningless for ksh since it will never be reached.
<P>
I've seen plenty of bash scripts with similar constructs where the author coded a quick 'exit' line into the while loop but forgetting to check the result, probably assuming the ksh behaviour.Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-12458201585883278932020-02-04T15:33:00.000+01:002020-02-04T15:33:58.757+01:00Circular shift of lists in python<h1>Scenario</h1>
Assume I have a <B>list</B> in python
<PRE>
a = [ -2, -1, 0, 1, 2 ]
</PRE>
(an admittedly simple list here with integer numbers).
<BR>
I want to <B>shift</B> these entries in the list to the left or right so that the entries moving out of the list enter the list again at the other end e.g. I want to <I>shift 2 to the left</I> to get this list:
<PRE>
[0, 1, 2, -2, -1]
</PRE>
<h1>Solutions</h1>
There are solutions using <B>numpy</B> but I want to present another easy solution just using <B>slices</B>.
<PRE>
shift = 2
x = a[shift:] # This is [ 0, 1, 2 ]
y = a[:shift] # This is [ -2, -1 ]
print( x + y )
[0, 1, 2, -2, -1]
</PRE>
I am
<LI>setting my shift value to <I>2</I>
<LI>creating a slice from index <I>2</I> to the end of the list
<LI>creating a slice from the beginning of the list until index - 1
<LI>adding the two slices to get the shifted result
<P>
This also works for negative shift values (shift to the right) which is a particularily nice feature of the pythong <B>[:]</B> operator.
<PRE>
shift = -1
x = a[shift:] # This is [ 2 ]
y = a[:shift] # This is [ -2, -1, 0, 1 ]
print( x + y )
[2, -2, -1, 0, 1]
</PRE>
<h1>Add-on: calculate the new index of a shifted element</h1>
Starting with
<PRE>
a = [ -2, -1, 0, 1, 2 ]
</PRE>
the element <I>-2</I> has index <I>0</I>.
The <I>shift 2 to the left</I> result
<PRE>
[0, 1, 2, -2, -1]
</PRE>
puts element <I>-2</I> at index <I>3</I>.
<P>
How can I calculate that?
<PRE>
def index_after_shift( old_index, shift ):
return ( old_index - shift ) % len(a)
# Examples
for shift in [2, -1 , 6 ]:
for ind in [ 0,1,2,3,4]:
print( ind, shift, index_after_shift( ind, shift ) )
print()
0 2 3
1 2 4
2 2 0
3 2 1
4 2 2
0 -1 1
1 -1 2
2 -1 3
3 -1 4
4 -1 0
0 6 4
1 6 0
2 6 1
3 6 2
4 6 3
</PRE>
The function <I>index_after_shift</I> calculates the new index.
<BR>
A <I>shift 2 to the left</I> i.e. <noformat>shift = 2</noformat> means that we <B>subtract</B> <I>2</I> from the current index to get to the new one therefore <noformat> old_index - shift</noformat>. This is easily understandable for indexes <I>2, 3, 4, ...</I> which will become <I>0, 1, 2, ...</I>.
<BR>
What do we do with smaller indexes?>
<BR>
Here we are using the <B>modulo</B> function which will do the necessary calculation for us e.g.
<PRE>
shift = 2
old_index = 1
# length of a is 5
( old_index - shift ) % len(a)
# = ( 1 - 2 ) % 5
# = -1 % 5
# = 4
</PRE>
The <B>modulo</B> has converted the negative number resulting from the subtraction into a positive one.
<P>
<I>shift to the right</I> means we have to <B>add</B> something to the index to get to our new index (since the variable <I>shift</I> is negative for right shifts we subtract a negative number in the function which results in the addition of a positive number).Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-21694544591874242202020-01-30T15:21:00.000+01:002020-01-30T15:21:48.249+01:00Modulo operation in programming languages - differently implementedMany programming languages support an operation which they call <B>modulo operator</B> and which is often designated by the symbol
<UL <B>%</B></UL>
Often it is also referred to as the <B>remainder after division</B>.
<P>
Only recently I found out that this operation does not behave as one might think. The results differ depending on which <B>programming language</B> you are using.
<P>
Before I go into the details I would like to illustrate the difference which shows when using <B>negative numbers</B>.
<H1>Example</H1>
I want to calculate these two expressions:
<PRE>
27 % 10
-27 % 10
</PRE>
The result for the second expression will be different depending on the programming language.
<h2>C</h2>
When you are using this statement in <B>C</B>
<PRE>
printf("%d %d\n", 17 % 10, -17 % 10 );
</PRE>
you get
<PRE>
7 -7
</PRE>
<h2>Python</h2>
When you are using this statement in <B>python</B>
<PRE>
print( '{0} {1}'.format( 17 % 10, -17 % 10 ) )
</PRE>
you get
<PRE>
7 3
</PRE>
<h1>Explanation</h1>
<h2>modulo as remainder by division</h2>
Some programming languages implement <B>modulo</B> as a <B>remainder of division</B> operation.
Thus <i>-27 % 10</i> results in the leftover of <i>-27</i> when you take away the maximum multiple of <i>10</i> , so you are left with <i>-7</i>.
<h2>modulo as <I>mathematically</I> correct number</h2>
Other programming languages implement <B>modulo</B> as correct in the mathematical sense.
<BR>
Mathematically modulo is defined as the number which needs to be added to a multiple of the divisor to get to the original.
<PRE>
x = m % n
There must be a number 'a' so that
a * n + x = m
and this condition should be met:
0 <= x < n
</PRE>
In our case:
<PRE>
x = 3
a = -2
=>
a * 10 + x = -2 * 10 + 3 = -17
</PRE>
<h1>Conclusion</h1>
Since I am not a programming languages expert I can only refer to the interesting Wikipedia article about <a href="https://en.wikipedia.org/wiki/Modulo_operation">modulo operations</a>.
<BR>
This subject is worth knowing if any of your programming efforts involve some number operations.
<BR>
My personal "watch out" topic is <I>awk</I> programming where the behaviour is non-mathematical like C.
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-62767236015946798082019-11-20T00:51:00.000+01:002019-11-20T00:51:18.038+01:00Ansible - dictionaries vs. listsWhen working with more complex variables in <B>Ansible</B> (often based on yaml input files) you always have to be aware whether the variable is a <B>list</B> or a <B>dictionary</B>. Applicable <B>filters</B> and <B>methods</B> differ and can lead to errors or unexpected results.
<H1>Example input yaml</H1>
Here I am defining two variables <I>x</I> and <I>y</I> with some sub elements.<BR>
At first glance there doesn't seem to be much difference and if you are about to design a yaml for whatever purpose both solutions might seem interchangeable.
The difference lies in its usage which we will see below.
<PRE>
x:
b1:
c1: 1
c2: "aaa"
c3:
b2:
c2: "bbb"
c3: 5
y:
- b1:
c1: 1
c2: "aaa"
c3:
- b2:
c2: "bbb"
c3: 5
</PRE>
Call the file <I>dict.yml</I>.
<h1>How to check the variable type</h1>
Lately there is a new filter in Ansible called <B>type_debug</B> which I find incredibly useful when in doubt.
<BR>(unfortunately it was not available in Ansible 1.x, it would have saved me a lot of time)
<PRE>
- hosts: localhost
tasks:
- include_vars: dict.yml
- debug:
msg: "x: {{x | type_debug}} / y: {{y |type_debug}}"
</PRE>
will show
<PRE>
TASK [debug] ************************************************************************************
ok: [localhost] => {
"msg": "x: dict / y: list"
}
</PRE>
i.e. I have a dictionary and a list.
<h1>How to access the elements</h1>
The elements of
<LI> a <B>dictionary</B> are accessed by <I>name</i> i.e. <I>x['b1']</I>
<LI> a <B>list</B> are accessed by <I>position</I> i.e. <I>y[0]</I>
<BR>
Something like <I>x[0]</I> or <I>y['b1']</I> would generate a <I>VARIABLE IS UNDEFINED</I>.
<BR>
I also show the variable type of the sub elements.
<PRE>
- hosts: localhost
tasks:
- include_vars: dict.yml
- debug:
msg: "{{x['b1'] | type_debug}}"
- debug:
var: x['b1']
- debug:
msg: "{{y[0] | type_debug}}"
- debug:
var: y[0]
</PRE>
will show
<PRE>
TASK [debug] ************************************************************************************
ok: [localhost] => {
"msg": "dict"
}
TASK [debug] ************************************************************************************
ok: [localhost] => {
"x['b1']": {
"c1": 1,
"c2": "aaa",
"c3": null
}
}
TASK [debug] ************************************************************************************
ok: [localhost] => {
"msg": "dict"
}
TASK [debug] ************************************************************************************
ok: [localhost] => {
"y[0]": {
"b1": {
"c1": 1,
"c2": "aaa",
"c3": null
}
}
}
</PRE>
You should also note the difference in the result. They are <B>both</B> dictionaries but in the x-case we get a simple dictionary with 3 elements whereas in the y-case we get a dictionary with one element <I>b1</I> which a sub element of type dictionary.
<h1>How to loop through sub elements</h1>
You can loop easily through the elements by supplying the variable to <I>with_items</I>.
The distinction is in what you get as an <B>item</B>.
<I>with_item</I> provides
<LI> <B>dictionary</B> elements as <B>strings</B> and you need to access the sub elements via the <I>{{x[item]}}</i> method
<LI> <B>list</B> elements are dictionaries
<PRE>
- hosts: localhost
tasks:
- include_vars: dict.yml
- debug:
msg: "{{item}}: {{item|type_debug}} / {{x[item]}}: {{x[item]|type_debug}}"
with_items: "{{x}}"
- debug:
msg: "{{item}}: {{item|type_debug}}"
with_items: "{{y}}"
</PRE>
<PRE>
TASK [debug] ************************************************************************************
ok: [localhost] => (item=b1) => {
"msg": "b1: AnsibleUnsafeText / {u'c3': None, u'c2': u'aaa', u'c1': 1}: dict"
}
ok: [localhost] => (item=b2) => {
"msg": "b2: AnsibleUnsafeText / {u'c3': 5, u'c2': u'bbb'}: dict"
}
TASK [debug] ************************************************************************************
ok: [localhost] => (item={u'b1': {u'c3': None, u'c2': u'aaa', u'c1': 1}}) => {
"msg": "{u'b1': {u'c3': None, u'c2': u'aaa', u'c1': 1}}: dict"
}
ok: [localhost] => (item={u'b2': {u'c3': 5, u'c2': u'bbb'}}) => {
"msg": "{u'b2': {u'c3': 5, u'c2': u'bbb'}}: dict"
}
</PRE>
<h1>How to access the bottommost elements</h1>
Say we want to access the value of <I>c2</I> of <I>b1</i> for both x and y.
There is in both cases the bracket and the dot approach for the dictionary sub elements.
In the y-case you need to supply the list position too but it also can be used with the dot approach.
<PRE>
- hosts: localhost
tasks:
- include_vars: dict.yml
- debug:
var: x['b1']['c2']
- debug:
var: x.b1.c2
- debug:
var: y[0]['b1']['c2']
- debug:
var: y.0.b1.c2
</PRE>
will lead to
<PRE>
TASK [debug] ************************************************************************************
ok: [localhost] => {
"x['b1']['c2']": "aaa"
}
TASK [debug] ************************************************************************************
ok: [localhost] => {
"x.b1.c2": "aaa"
}
TASK [debug] ************************************************************************************
ok: [localhost] => {
"y[0]['b1']['c2']": "aaa"
}
TASK [debug] ************************************************************************************
ok: [localhost] => {
"y.0.b1.c2": "aaa"
}
</PRE>
<h1>Conclusion</h1>
Filters like <I>join, unique, setaddr, map etc.</I> only make sense for the correct variable type.
You always need to know what you are dealing with and thus you will be able to create working playbooks faster.
It might also influence your design decision when you want to map data into a fitting yaml.
<h1>Why did I write this article</h1>
When I am writing Ansible playbooks they are often based on <B>input yaml files</B> (sometimes my own, sometimes from others) and also often these yaml files contain complex structures which need to be parsed and interpreted correctly.
<BR>
I am parsing complex structures by creating intermediate steps and creating new variables with <B>set_fact</B> which contain sub structures of the original one. A common mistake I make is that at certain points in my code I am not sure whether the variable in use is a <B>list</B> or a <B>dictionary</B>. Subsequently when I am using a method or filter this can lead to an error or - worse - to a valid but incorrect result (e.g. an "empty" variable) which will lead to further content errors downstream and the end result is puzzling.
<BR>So I thought for my sake and the sake of the reader a little summary article would help, in particular since I find the Ansible documentation not always as helpful as it could be.Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com12tag:blogger.com,1999:blog-7194384172575329109.post-79691304254266650732019-10-15T12:38:00.002+02:002019-10-15T16:29:42.446+02:00Identifying and handling Java keystore store types in different Java versionsRecently I encountered a different behaviour of <B>Java keytool</B> in newer versions of Java (I tested 10.0.2 and 11.0.4) compared to older versions (I tested 7.0_131 and 8.0_144) in regards to <B>identifying store types.</B>
<P>
<h3>Generate keystore with <U>various</U> store types</h3>
<I>keytool</I> is able to handle three different store types.<BR>
Basically I am running these commands (regardless which Java version is in place).
Store types are not case sensitive.
<PRE>
keytool -genkey -alias foo -keystore a.keystore -storetype JKS
keytool -genkey -alias foo -keystore b.keystore -storetype JCEKS
keytool -genkey -alias foo -keystore c.keystore -storetype PKCS12
</PRE>
<P>
In detail:<BR>
I supply a password and simply hit <I>Enter</I> to all the questions being asked except the last one (answer <I>y</I>). I also don't provide a different key password.<BR>
Note that the default store type has also changed between Java versions (JKS in Java 8, PKCS12 in Java 10) so running the above commands without <i>-storetype ..</I> will generate keystores of different type resp. to your Java version in use. It is best to always indicate a store type in order to avoid any disambiguity.
<PRE>
keytool -genkey -alias foo -keystore a.keystore -storetype jks
Enter keystore password:
Re-enter new password:
What is your first and last name?
...
Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown correct?
[no]: y
Enter key password for <a.keystore>
(RETURN if same as keystore password):
</PRE>
<h3>Checking the keystore type via <I>keytool -list</I></h3>
Since <I>keytool -list ...</I> also indicates the <B>store type</B> (before listing the aliases) I am running
<PRE>
keytool -list -keystore a.keystore
keytool -list -keystore b.keystore
keytool -list -keystore c.keystore
</PRE>
<h4><font color=red>Result for Java 7 and 8</font></h4>
<PRE>
keytool -list -keystore a.keystore
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
foo, Oct 15, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): 69:B0:E2:F8:D4:C3:77:70:A2:20:AC:BE:51:B1:6A:FF:71:85:AF:4F
</PRE>
<PRE>
keytool -list -keystore b.keystore
keytool error: java.io.IOException: Invalid keystore format
keytool -list -keystore c.keystore
keytool error: java.io.IOException: Invalid keystore format
</PRE>
<P>
i.e. <FONT color=red><I>keytool</I> does not recognize the store type in Java 7 and 8</FONT>
<BR>If you want to see the list of aliases you <B>need to know the store type in advance and indicate it on the command line.</B>
<PRE>
keytool -list -keystore b.keystore -storetype JCEKS
Enter keystore password:
Keystore type: JCEKS
Keystore provider: SunJCE
Your keystore contains 1 entry
foo, Oct 15, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): 87:08:88:32:20:F2:61:11:C3:82:E0:DF:26:76:7B:8A:CD:2C:2C:B8
</PRE>
<PRE>
keytool -list -keystore c.keystore -storetype PKCS12
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SunJSSE
Your keystore contains 1 entry
foo, Oct 15, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): E8:C4:B1:F2:E2:C9:7F:9F:D0:10:77:3D:F5:07:30:77:3C:E3:74:8D
</PRE>
<h4><font color=red>Result for Java 10 and 11</font></h4>
The newer Java versions are able to list any store type.
<PRE>
keytool -list -keystore a.keystore
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
foo, Oct 15, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 25:EB:EE:65:AC:AD:F5:44:58:90:82:E3:BE:9D:A2:AD:D0:EC:91:50:CB:72:75:C9:03:12:0F:58:4F:A5:FA:44
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore x.keystore -destkeystore x.keystore -deststoretype pkcs12".
</PRE>
<PRE>
keytool -list -keystore b.keystore
Enter keystore password:
Keystore type: JCEKS
Keystore provider: SunJCE
Your keystore contains 1 entry
foo, Oct 15, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA-256): B2:61:07:4D:D5:12:58:D7:F8:49:BB:1A:7E:69:FC:F2:C4:3E:3A:9B:F3:B0:E0:F0:F5:3E:9F:46:C4:8B:A7:86
Warning:
The JCEKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore y.keystore -destkeystore y.keystore -deststoretype pkcs12".
</PRE>
<PRE>
keytool -list -keystore c.keystore
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 1 entry
foo, Oct 15, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA-256): B4:E3:38:34:21:27:AC:0F:43:09:46:BC:FE:41:DF:1E:F5:5F:3A:75:75:52:D3:7C:42:A4:F7:57:B5:FC:34:9E
</PRE>
<P>
Note that the <B>fingerprints</B> are different and the store types <B>JKS and JCEKS</B> generate a warning.<BR>
The <B>keystore provider</B> for <B>PKCS12</B> is <B>SunJSSE</B> in Java 7 and <B>SUN</B> in Java 10.
<hr>
On <B>UNIX</B> systems you can use the <B>file</B> command to get an indication of the resp. store type.
<PRE>
file *.keystore
a.keystore: Java KeyStore
b.keystore: Java JCE KeyStore
c.keystore: data
</PRE>
i.e. the file "magic" can identify the old JKS and JCEKS keystores but the newest recommended PKCS12 is only shown as <I>data</I>.
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-16641133197665294122019-09-19T13:42:00.000+02:002019-09-19T15:17:28.275+02:00Assignment operator in python vs. copyLately I stumbled across the following issue when assigning a variable to another (which was a <B>list</B>) and a change to the original reflected in the second.
<BR>
(<I>This is in python 3.7. I am not using the python prompt <B>>>></B> in the examples below.</I>)
<P>
<pre>
x = ['a']
y = x
x.append('b')
print(x)
['a', 'b']
print(y)
['a', 'b']
</PRE>
<P>
An example with <B>int</B> shows this
<PRE>
x = 5
y = x
x = x + 7
print(x)
12
print(y)
5
</PRE>
<P>
What might seem surprising at first glance totally makes sense:
<BR>
<B>an assignment <font color=red>'='</font> is <font color=red>not</font> a <font color=red>copy</font> operation.</B>
<P>
The assignment
<PRE>
x = something
</PRE>
rather gives the name tag <B>x</B> to the <B>object <I>something</I></B>.
<BR>
<PRE>
y = x
</PRE>
gives the name tag <B>y</B> to the <B>same object</B> which is tagged <B>x</B>.
<BR>
There is no copy aka. creation of a new object happening here.
<P>
The conclusion is:
<BR>
if the object tagged <B>x</B> is a <I>changeable object</I> any change will be seen in <B>y</B> too.<BR>
So if the object is a list, tuple, set, dict etc. a change to the object (like the <I>append</I> operation) is both visible when either accessing <B>x</B> or <B>y</B>.
<P>
If the object is an integer or a string then it cannot be changed. Operations like addition create a <B>new object</B>.<BR>
<I>x = x + 7</I> does not change the former object <B>5</B> but creates a new object <B>12</B>.<BR>
<B>y</B> is still pointing to the old object and thus shows the old value.
<P>
If the intention of the original code was <B>to create a copy of a list</B> then a true copy operation should be used instead.
<PRE>
y = x[:]
</PRE>
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-71407425210873744602019-03-06T16:48:00.000+01:002019-09-19T12:54:16.072+02:00How to determine the number of elements of each directory in a directory treeImagine that you have a directory tree with multiple sub directories and files and you want to determine the number of files and directories for each directory in this hierarchy i.e. not just <B>direct</B> elements but all elements of sub directories as well.
Here I present a solution just using the UNIX tools <B>find</B> and <B>awk</B>.
<P>
<h1>Creating an example directory tree</H1>
With these commands I am creating a simple directory tree with some directories and files which I will use further on.
<PRE>
mkdir -p somedir/d1/b somedir/d1/c somedir/d2 somedir/d3; # Create some directories
touch somedir/d1/b/file1 somedir/d3/file2; # Create some files
ln -s somedir/d3/file2 somedir/d3/z; # Create symbolic links
(cd somedir; ln -s d1 d4_link ;)
find somedir
somedir
somedir/d1
somedir/d1/b
somedir/d1/b/file1
somedir/d1/c
somedir/d2
somedir/d3
somedir/d3/file2
somedir/d3/z
somedir/d4_link
</PRE>
I want some code to show me that <B>somedir</B> contains 9 elements and <B>somedir/d1</B> contains 3 elements aso.
<h1>List the complete directory tree with types of files</H1>
On Linux systems with <B>GNU find</B> (<I>unfortunately this does not work on my Mac</I>) one can easily retrieve a list of elements and their file types with the <B>printf</B> formatting options.
My example contains six directories (including the topmost <I>somedir</I>), two files and two symbolic links (one to a file and one to a directory).
<PRE>
find somedir -printf '%y %p\n'
d somedir
d somedir/d1
d somedir/d1/b
f somedir/d1/b/file1
d somedir/d1/c
d somedir/d2
d somedir/d3
f somedir/d3/file2
l somedir/d3/z
l somedir/d4_link
</PRE>
<h1>Calculating the number of entries per directory</h1>
First of all the <B>awk</B> command consumes a slightly modified output of the <B>find</B> from above:<BR>
I am adding a <B>slash</B> so that I can use the slash as an awk field delimiter.
<PRE>
find somedir -printf '%y/%p\n'
...
d/somedir/d1/b
...
</PRE>
The awk command:
<PRE>
awk -F/ '
{
x = $2;
count[x]++;
for(i=3;i<=NF;i++) { x = x FS $i; count[x]++ }
type[x] = $1;
}
END {
for(i in count) printf "%s %2d %s\n", type[i],count[i]-1,i
}'
</PRE>
What happens in each step: the first field is the type of the element ( d for directory, f for field etc.), the count array is increased for each occurance of a path.
<PRE>
d/somedir
x = "somedir"
count["somedir"] = 1
# the for loop is not executed for "d/somedir" since NF=2
type["somedir"} = "d"
d/somedir/d1
x = "somedir"
count["somedir"] = 2 # increase count by 1
# NF = 3. The for loop is executed once.
x = x FS "a" = "somedir/d1"
count["somedir/d1"] = 1 # first time: 1
type["somedir/d1"] = "d"
d/somedir/d1/b
x = "somedir"
count["somedir"] = 3 # increase count by 1
# NF = 4. The for loop is executed twice.
x = x FS "a" = "somedir/d1"
count["somedir/d1"] = 2 # increase count by 1
x = x FS "b" = "somedir/d1/b"
count["somedir/d1/b"] = 1 # first time: 1
type["somedir/d1/b"] = "d"
</PRE>
<P>
Here is the combined command sequence also appended by a <B>sort</B> statement for better readability
<PRE>
find somedir -printf '%y/%p\n' |
awk -F/ '{ x=$2; count[x]++; for(i=3;i<=NF;i++) { x=x FS $i; count[x]++ } type[x]=$1;}
END {for(i in count) printf "%s %2d %s\n", type[i],count[i]-1,i } '|
sort -k3,3
d 9 somedir
d 3 somedir/d1
d 1 somedir/d1/b
f 0 somedir/d1/b/file1
d 0 somedir/d1/c
d 0 somedir/d2
d 2 somedir/d3
f 0 somedir/d3/file2
l 0 somedir/d3/z
l 0 somedir/d4_link
</PRE>
So <B>somedir</B> contains 9 elements altogether: 4 direct elements <B>d1, d2, d3 and d4_link</B> and also elements of elements.
<BR>
Note that the <B>END</B> statement prints the count minus one since the count was set to one when the directory appeared originally but we want to show only the number of elements i.e. I need to exclude the directory itself.<BR>
Note also that <B>somedir/d4_link</B> (the symbolic link to directory <I>somedir/d1</i>) is not followed and listed as having <B>zero</B> elements.
If you want to follow symbolic links to directories with <B>find somedir -follow</B> the calculations will be misleading since - in this example - elements of <B>d1</B> and <B>d4_link</B> would be calculated twice.
<P>
The counts for non-directory file types should always be zero, they could probably be excluded completely from the output.
<PRE>
find somedir -printf '%y/%p\n' |
awk -F/ '{ x=$2; count[x]++; for(i=3;i<=NF;i++) { x=x FS $i; count[x]++ } type[x]=$1;}
END {for(i in count) if( type[i]=="d") printf "%2d %s\n", count[i]-1,i } '| sort -k2,2
9 somedir
3 somedir/d1
1 somedir/d1/b
0 somedir/d1/c
0 somedir/d2
2 somedir/d3
</PRE>
<h1>Usages</h1>
<h2>Empty directories</h2>
Add a <B>grep '^d 0'</B> (or adjust the awk code with if clause count[i]==1 etc.)
<PRE>
find somedir -printf '%y/%p\n' |
awk -F/ '{ x=$2; count[x]++; for(i=3;i<=NF;i++) { x=x FS $i; count[x]++ } type[x]=$1;}
END {for(i in count) printf "%s %2d %s\n", type[i],count[i]-1,i } '|
grep '^d 0'
d 0 somedir/d1/c
d 0 somedir/d2
</PRE>
<h2>Non-empty directories</h2>
<PRE>
find somedir -printf '%y/%p\n' |
awk -F/ '{ x=$2; count[x]++; for(i=3;i<=NF;i++) { x=x FS $i; count[x]++ } type[x]=$1;}
END {for(i in count) printf "%s %2d %s\n", type[i],count[i]-1,i } '|
grep '^d .[^0]' | sort -k 3,3
d 9 somedir
d 3 somedir/d1
d 1 somedir/d1/b
d 2 somedir/d3
</PRE>
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-48209136223203680312019-02-20T20:03:00.001+01:002019-02-20T20:19:10.876+01:00Using the 'timeout' commandIn Linux there is <B>timeout</B> command (usually in /usr/bin/timeout) which is very handy when you really want to restrict possibly long running commands.
Here I want to show a couple of invocations to explain its usage.
<P>
First of all: <B>timeout</B> exits with error code <B>124</B> if the timeout has been reached.
<BR>
Of particular interest is the handling of exit codes if commands are failing.
<P>
I am playing with various invocations of a timeout setting of <B>3 seconds</B> and a sleep command of <B>5 seconds</B> and vice versa.
<TABLE BORDER>
<TR>
<TH>
</TH>
<TH>Output</TH>
<TH>Exit code</TH>
</TR>
<TR><TD COLSPAN=4>The timeout threshold is reached.</TD></TR>
<TR>
<TD><PRE>timeout 3 sleep 5</PRE></TD><TD></TD><TD>124</TD><TD></TD>
</TR>
<TR>
<TD><PRE>timeout 3 sh -c "sleep 5; exit 1"</PRE></TD><TD></TD><TD>124</TD><TD></TD>
</TR>
<TR>
<TD><PRE>timeout 3 sh -c "echo something; sleep 5; exit 1"</PRE></TD><TD>something</TD><TD>124</TD><TD>Failure<BR>but part of the commands ran successfully creating output.</TD>
</TR>
<TR>
<TD><PRE>timeout 3 sh -c " sleep 5; echo something; exit 1"</PRE></TD><TD></TD><TD>124</TD><TD></TD>
</TR>
<TR><TD COLSPAN=4>The timeout threshold is not reached.</TD></TR>
<TR>
<TD><PRE>timeout 5 sleep 3</PRE></TD><TD></TD><TD>0</TD><TD>Success.<BR>The command finished before the timeout threshold.</TD>
</TR>
<TR>
<TD><PRE>timeout 3 sh -c "exit 1"</PRE></TD><TD></TD><TD>1</TD><TD>Failure.<BR>The command finished before the timeout threshold but with an error.</TD>
</TR>
<TR>
<TD><PRE>timeout 3 sh -c "echo something; exit 1"</PRE></TD><TD>something</TD><TD>1</TD><TD>Failure.<BR>The commands ran and finished before the timeout threshold but with an error.</TD>
</TR>
</TABLE>
<P>
Here is an example that shows how a loop gets executed partially.
<PRE>
timeout 3 sh -c "for i in 1 2 3; do sleep 1; echo i=\$i; done"
i=1
i=2
and exit code 124
</PRE>
In this case one needs to carefully continue and understand possible consequences of a partially executed command chain and how to recover from timeout errors.
<P>
<B>Error handling</B> could be done this way:
<PRE>
case $? in
0)
# all ok
;;
124)
# this was a timeout
;;
*)
# some other error induced by the executed command(s)
;;
esac
</PRE>
<P>
Maybe this usage is also helpful sometimes: capture only the timeout exit code.
<PRE>
timeout 3 sh -c "echo something; sleep 5" || { [ $? -eq 124 ] && echo TIMEOUT; }
echo $?
will show output of the executed command, the timeout exit code check and an exit code 0
something
TIMEOUT
0
</PRE>
Interesting to see what happens when the executed command fails. The <B>exit code of the failed command</B> is still available to subsequent checks.
<PRE>
timeout 3 sh -c "echo something; exit 1" || { [ $? -eq 124 ] && echo TIMEOUT; }
echo $?
will show output of the executed command(s) and the exit code of the command chain
something
1
</PRE>
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-48904180722372095532019-02-14T21:44:00.001+01:002019-02-19T00:55:55.689+01:00Ansible tags with 'include'When using the <B>include</B> directive things do not quite work as I would expect when I also add a tag to the include section.
<P>Note: this was tested with Ansible 2.5.2 and python 2.7.15
<h1>Scenario</h1>
I split the playbook of the previous example into two:
<BR>The tasks (tagged in the same fashion as before) have been moved to a new file <noformat>include_this.yml</noformat>:
<PRE>
---
- name: Debug No Tag
debug:
msg: "No tag"
- name: Debug Tag A
debug:
msg: "Tag A"
tags:
- tagA
- name: Debug Tag B
debug:
msg: "Tag B"
tags:
- tagB
- name: Debug Tag A and B
debug:
msg: "Tag A and B"
tags:
- tagA
- tagB
</PRE>
The playbook <noformat>playbook.yml</noformat> has been reduced to two tasks:
<UL>
<LI>one debug task
<LI>one <I>include</I> task
</UL>
I am testing two versions: the second one has an <B>additional tag for the include directive</B>.<P>
<TABLE><TR>
<TH>Version 1</TH><TH>Version 2</TH></TR><TR>
<TD valign=top>
<PRE>
- name: Debug Playbook
hosts: localhost
tasks:
- name: Debug No Tag in Playbook
debug:
msg: "No tag in playbook"
- name: Include
include: include_this.yml
</PRE>
</TD>
<TD>
<PRE>
- name: Debug Playbook
hosts: localhost
tasks:
- name: Debug No Tag in Playbook
debug:
msg: "No tag in playbook"
- name: Include
include: include_this.yml
tags:
- tagA
</PRE>
</TD>
</TR>
</TABLE>
<P>
The playbook is run as <PRE>ansible-playbook [--tags tagA] [--skip-tags tagB] playbook.yml</PRE>
<P>
<h1>Results for version 1</h1>
If called with <I>--tags</I> or <I>--skip-tags</I> version 1 delivers the same results as described in my previous post when the include file was part of the playbook.
<BR>
The additional 5th task in playbook.yml is executed in case of <I>--skip-tags tagB</I> or when no arguments are supplied.
<TABLE>
<TR>
<TD>no args</TD><TD> <PRE>TASK [Debug No Tag in Playbook]
TASK [Debug No Tag]
TASK [Debug Tag A]
TASK [Debug Tag B]
TASK [Debug Tag A and B]
</PRE></TD>
</TR>
<TR>
<TD><I>--tags tagA</I></TD><TD> <PRE>
TASK [Debug Tag A]
TASK [Debug Tag A and B]
</PRE></TD>
</TR>
<TR>
<TD><I>--tags tagA --skip-tags tagB</I></TD><TD> <PRE>
TASK [Debug Tag A]
</PRE></TD>
</TR>
<TR>
<TD><I>--skip-tags tagB</I></TD><TD> <PRE>TASK [Debug No Tag in Playbook]
TASK [Debug No Tag]
TASK [Debug Tag A]
</PRE></TD>
</TR>
<TR><TD COLSPAN=3>Now I am reversing tagA and tagB in the call which leads to the expected results.</TD></TR>
<TR>
<TD><I>--tags tagB</I></TD><TD> <PRE>
TASK [Debug Tag B]
TASK [Debug Tag A and B]
</PRE></TD>
</TR>
<TR>
<TD><I>--tags tagB --skip-tags tagA</I></TD><TD> <PRE>
TASK [Debug Tag B]
</PRE></TD>
</TR>
</TABLE>
<h1>Results for version 2</h1>
<TABLE>
<TR>
<TD>no args</TD><TD> <PRE>TASK [Debug No Tag in Playbook]
TASK [Debug No Tag]
TASK [Debug Tag A]
TASK [Debug Tag B]
TASK [Debug Tag A and B]
</PRE></TD>
<TD>Same as version 1.<BR>
<B>All five</B> tasks are being executed for both version of the playbook. This works as expected. No restrictions of any sort apply.</TD>
</TR>
<TR>
<TD><I>--tags tagA</I></TD><TD> <PRE>
TASK [Debug No Tag]
TASK [Debug Tag A]
TASK [Debug Tag B]
TASK [Debug Tag A and B]
</PRE></TD>
<TD>
All <B>four</B> task of the included files are being executed. <B>No filtering by tagA for the included file takes place.</B> This was a complete suprise to me. It seems that the tagging of the include step supersedes somehow the tags of the included file.
</TD>
</TR>
<TR>
<TD><I>--tags tagA --skip-tags tagB</I></TD><TD> <PRE>
TASK [Debug No Tag]
TASK [Debug Tag A]
</PRE></TD>
<TD>
The negative filter is applied to the result before and leaves two tasks for execution.
</TD>
</TR>
<TR>
<TD><I>--skip-tags tagB</I></TD><TD> <PRE>TASK [Debug No Tag in Playbook]
TASK [Debug No Tag]
TASK [Debug Tag A]
</PRE></TD>
<TD>
Same as version 1.<BR>The two tasks with tagB are skipped and the other three are executed.
</TD>
</TR>
<TR>
<TD><I>--tags tagB</I></TD><TD> <PRE>
TASK [Debug Tag B]
TASK [Debug Tag A and B]
</PRE></TD>
<TD>
Same as version 1.<BR>The two tasks with tagB are executed and the tag setting in the include directive is not taken into account.
</TD>
</TR>
<TR>
<TD><I>--tags tagB --skip-tags tagA</I></TD><TD> <PRE>... nothing ...</PRE>
</TD>
<TD>
Now here is a surprise. The <I>--skip-tags tagA</I> has skipped the whole include file and no task is being executed at all.
</TD>
</TR>
</TABLE>
<h1>An explanation using sets</h1>
Let's look at it from a <B>set</B> perspective.
<TABLE>
<TR>
<TD><I>--tags tagA</I></TD><TD><I>--skip-tags tagA</I></TD><TR>
<TR><TD COLSPAN=2>These two results are complementary and - if joined - build the complete set.</TD></TR>
<TR>
<TD><PRE>
TASK [Debug No Tag]
TASK [Debug Tag A]
TASK [Debug Tag B]
TASK [Debug Tag A and B]
</PRE>
</TD>
<TD VALIGN=TOP>
<PRE>TASK [Debug No Tag in Playbook] </PRE>
</TD>
</TR>
<TR><TD COLSPAN=2>Here the same complimentary sets for tagB.</TD></TR>
<TR>
<TD><I>--tags tagB</I></TD><TD><I>--skip-tags tagB</I></TD><TR>
<TR>
<TD VALIGN=TOP><PRE>
TASK [Debug Tag B]
TASK [Debug Tag A and B]
</PRE>
</TD>
<TD>
<PRE>TASK [Debug No Tag in Playbook]
TASK [Debug No Tag]
TASK [Debug Tag A] </PRE>
</TD>
</TR>
</TABLE>
Now any invocation of <I>--skip-tags</I> leads to an <B>intersection</B> with the respective sets.<BR>
Example: <I>--tags tagB --skip-tags tagA</I>: the sets have no task in common and thus nothing will be executed.
<h1>Conclusion</h1>
My idea was that a tag for the include directive would determine <B>whether</b> the include happens or not. Obviously wrong. It seems that it only determines which other tasks in the playbook.yml are executed i.e. it prohibits the untagged task from being run.
The include takes place in any case (see the examples when using tagB either in <I>--tags or --skip-tags</I>) but in a strange way since only other tags than the supplied one are applied.
I find this hard to remember or explain so my personal rule of mutual tag exclusion will be:
<UL>
<LI>tagged include section => no tags in the included file
<LI>tags in the included file => no tag for the include section
</UL>
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-30619847780101948562019-02-14T20:21:00.000+01:002019-02-14T21:44:54.935+01:00Ansible tagsLately I got a little confused about properly using <B>Ansible tags</B> so I thought I spend a couple of minutes to create a simple example to show how to use them and also explain the pitfalls (at least how I see them).
<P>
<P>Note: this was tested with Ansible 2.5.2 and python 2.7.15
<h1>Scenario</h1>
My example scenario is simple:
<UL>
<LI>I have a small playbook with <B>4 tasks</B>: one task without tag, two tasks with a different tag each and the 4th task with both tags
<PRE>
- name: Debug Playbook
hosts: localhost
tasks:
- name: Debug No Tag
debug:
msg: "No tag"
- name: Debug Tag A
debug:
msg: "Tag A"
tags:
- tagA
- name: Debug Tag B
debug:
msg: "Tag B"
tags:
- tagB
- name: Debug Tag A and B
debug:
msg: "Tag A and B"
tags:
- tagA
- tagB
</PRE>
<LI>I am running <B>4 invocations</B> of the playbook with a mix of <I>--tags<I> and </I>--skip-tags</I> to show the effects on the possible outcomes
</UL>
<h1>Results</h1>
<UL>
<LI>No tag arguments: <PRE>ansible-playbook playbook.yml</PRE>
<B>All</B> tasks are executed.
<PRE>
TASK [Debug No Tag] *****************************
ok: [localhost] => {
"msg": "No tag"
}
TASK [Debug Tag A] ******************************
ok: [localhost] => {
"msg": "Tag A"
}
TASK [Debug Tag B] ******************************
ok: [localhost] => {
"msg": "Tag B"
}
TASK [Debug Tag A and B] ************************
ok: [localhost] => {
"msg": "Tag A and B"
}
</PRE>
<LI>Supply <i>--tags tagA</I>: <PRE>ansible-playbook --tags tagA playbook.yml</PRE>
All tasks are <B><I>executed</I> where tagA is listed</B>.
This works as expected: <I>--tags</I> acts as a filter.
<PRE>
TASK [Debug Tag A] ******************************
ok: [localhost] => {
"msg": "Tag A"
}
TASK [Debug Tag A and B] ************************
ok: [localhost] => {
"msg": "Tag A and B"
}
</PRE>
<LI>Use both <I>--tags</I> and <I>--skip-tags</I>: <PRE>ansible-playbook --tags tagA --skip-tags tagB playbook.yml</PRE>
<B>Only</B> the task tagged <I>tagA</I> is executed. The task with both tags is skipped.
<BR>This actually got me confused. Now when writing this blog it seems obvious: <I>--skip-tags</I> acts as a negative filter i.e. it will skip all tasks where <I>tagB</I> is mentioned.
<PRE>
TASK [Debug Tag A] ******************************
ok: [localhost] => {
"msg": "Tag A"
}
</PRE>
<LI>Supply <I>--skip-tags tagB</I>: <PRE>ansible-playbook --skip-tags tagB playbook.yml</PRE>
All tasks are <B><I>skipped</I> where tagB is listed</B>, in particular the task without tags is also executed.
<PRE>
TASK [Debug No Tag] *****************************
ok: [localhost] => {
"msg": "No tag"
}
TASK [Debug Tag A] ******************************
ok: [localhost] => {
"msg": "Tag A"
}
</PRE>
</UL>
<h1>To remember</h1>
<UL>
<LI>
As soon as you switch to using tags via <I>--tags</I> <B>none</B> of the <B>untagged</B> tasks will be executed any longer.
If you want the untagged task to be executed in any case you need to <B>supply</B> all possible tags of your playbooks.
<LI>When mixing <I>--tags</I> and <I>--skip-tags</I> in one call both filters work in conjunction: <I>--tags</I> selects all tasks with the given tags(s), <I>--skip-tags</I> reduces this set by all tasks with the given skip tags.
</UL>
<HR>
The scenario can get a bit trickier when invoking <B>include</B> sections. I will explain this in another blog.Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-42746767153221227192019-02-14T19:32:00.003+01:002019-02-14T19:32:43.867+01:00A list of Google fonts Here is a list of <B>Google fonts</B>. It is constructed with Javascript and CSS and dynamically embedded into this blog.
<P>
Some fonts can be made more effective in bigger sizes or by applying font effects but that would be too much for this overview.<BR>
Enjoy and find <B>your</B> font.
<P>
<style>
#outerDIV {margin: 0; width: 100%; position: relative; background-color: lightgrey; color: black ; }
</style>
<P>
<div id="outerDIV" >
</div>
<script>
var fonts = ["Abel", "Abril Fatface", "Acme", "Aleo", "Alfa Slab One",
"Allan", "Amarante", "Amatic SC", "Anaheim", "Angkor",
"Anton", "Archivo Narrow", "Arima Madurai", "Arimo", "Arvo",
"Asap", "Atma", "Audiowide", "Autour One", "Averia Gruesa Libre",
"Averia Libre", "Averia Sans Libre", "Averia Serif Libre", "B612 Mono", "B612",
"Baloo Bhai", "Baloo Bhaijaan", "Baloo Bhaina", "Baloo Chettan", "Baloo Paaji",
"Baloo Tamma", "Baloo Tammudu", "Baloo Thambi", "Baloo", "Bangers",
"Baumans", "Bevan", "Bitter", "Black Ops One", "Boogaloo",
"Bowlby One SC", "Bowlby One", "Bubblegum Sans", "Bungee Inline", "Bungee Shade",
"Bungee", "Cabin Sketch", "Cabin", "Cairo", "Carter One",
"Catamaran", "Ceviche One", "Changa One", "Chango", "Chela One",
"Chelsea Market", "Cherry Cream Soda", "Cherry Swash", "Chewy", "Chicle",
"Chonburi", "Cinzel Decorative", "Coda", "Codystar", "Coiny",
"Comfortaa", "Concert One", "Contrail One", "Corben", "Courgette",
"Creepster", "Crimson Text", "Croissant One", "Crushed", "Cute Font",
"Dancing Script", "Diplomata SC", "Dosis", "Dynalight", "Eater",
"Elsie Swash Caps", "Elsie", "Emilys Candy", "Encode Sans Condensed", "Exo 2",
"Exo", "Expletus Sans", "Farsan", "Faster One", "Finger Paint",
"Fira Sans", "Fjalla One", "Fontdiner Swanky", "Forum", "Freckle Face",
"Fredericka the Great", "Fredoka One", "Frijole", "Fugaz One", "Galada",
"Galindo", "Germania One", "Glass Antiqua", "Gloria Hallelujah", "Graduate",
"Gravitas One", "Griffy", "Gruppo", "Gugi", "Hanalei Fill",
"Happy Monkey", "Heebo", "Henny Penny", "Hind Madurai", "Hind Siliguri",
"Hind", "Iceberg", "Iceland", "Inconsolata", "Indie Flower",
"Josefin Sans", "Joti One", "Kanit", "Karla", "Katibeh",
"Kavoon", "Kelly Slab", "Kirang Haerang", "Knewave", "Kranky",
"Lalezar", "Lato", "Lemon", "Lemonada", "Libre Baskerville",
"Libre Franklin", "Life Savers", "Lilita One", "Lily Script One", "Limelight",
"Lobster Two", "Lobster", "Londrina Outline", "Londrina Solid", "Lora",
"Love Ya Like A Sister", "Luckiest Guy", "Maiden Orange", "Major Mono Display", "McLaren",
"Medula One", "Megrim", "Merriweather Sans", "Merriweather", "Metamorphous",
"Milonga", "Mirza", "Mogra", "Monoton", "Montserrat",
"Mountains of Christmas", "Mukta", "Muli", "Mystery Quest", "Nanum Gothic",
"New Rocker", "Nixie One", "Nosifer", "Noto Sans JP", "Noto Sans KR",
"Noto Sans", "Noto Serif", "Nova Flat", "Nova Round", "Nova Script",
"Nova Slim", "Nova Square", "Nunito Sans", "Nunito", "Offside",
"Oldenburg", "Oleo Script Swash Caps", "Oleo Script", "Open Sans", "Oregano",
"Oregano", "Oswald", "Overlock SC", "Overlock", "Oxygen",
"PT Sans Narrow", "PT Sans", "PT Serif", "Pacifico", "Paprika",
"Passion One", "Patua One", "Peralta", "Permanent Marker", "Pirata One",
"Plaster", "Playball", "Playfair Display", "Poiret One", "Poller One",
"Pompiere", "Poor Story", "Poppins", "Press Start 2P", "Prosto One",
"Purple Purse", "Questrial", "Quicksand", "Racing Sans One", "Rakkas",
"Raleway Dots", "Raleway", "Rammetto One", "Ranchers", "Ranga",
"Revalia", "Ribeye", "Righteous", "Roboto Condensed", "Roboto Mono",
"Roboto Slab", "Roboto", "Rubik", "Ruslan Display", "Rye",
"Sail", "Salsa", "Sancreek", "Sarabun", "Sarina",
"Seaweed Script", "Shadows Into Light", "Share", "Shojumaru", "Shrikhand",
"Sigmar One", "Signika", "Simonetta", "Skranji", "Slabo 27px",
"Slackey", "Sniglet", "Sonsie One", "Source Code Pro", "Source Sans Pro",
"Special Elite", "Spicy Rice", "Spirax", "Squada One", "Srisakdi",
"Staatliches", "Stardos Stencil", "Stint Ultra Condensed", "Stint Ultra Expanded", "Teko",
"Thasadith", "Titan One", "Titillium Web", "Trade Winds", "Tulpen One",
"Ubuntu Condensed", "Ubuntu", "Uncial Antiqua", "Unica One", "UnifrakturMaguntia",
"Unkempt", "Vampiro One", "Varela Round", "Vast Shadow", "Viga",
"Voces", "Wallpoet", "Warnes", "Wellfleet", "Work Sans",
"Yanone Kaffeesatz", "Yatra One", "Yeon Sung", "Yeseva One", "ZCOOL KuaiLe",
"ZCOOL QingKe HuangYou", "ZCOOL XiaoWei" ]
function createStylesAndLinks( ) {
var len = fonts.length;
var style;
var styleName;
var link;
//len= 12
for( i=0; i<len; i++ ) {
styleName = "font" + i
styleName = "font" + i
style = document.createElement('STYLE');
style.innerHTML = "." + styleName + "{ font-family: '" + fonts[i] + "' ; font-size: 24pt ;}"
document.getElementsByTagName('head')[0].appendChild(style);
link = document.createElement('LINK' );
link.setAttribute( 'href', 'https://fonts.googleapis.com/css?family=' + encodeURI( fonts[i] ) );
link.setAttribute( 'rel', "stylesheet" ) ;
document.getElementsByTagName('head')[0].appendChild( link );
}
}
// This should be done only once when loading the web page
createStylesAndLinks();
function createDivs( parentDiv, textString ) {
var len = fonts.length;
var styleName;
var div;
var item;
for( i=0; i<len; i++ ) {
styleName = "font" + i
div = document.createElement('DIV');
div.setAttribute( "class", styleName );
div.innerHTML = "<BR><I>Font name:</I> " + fonts[i] + "<BR>" + textString ;
item = document.getElementById( parentDiv );
item.appendChild( div );
}
}
createDivs( "outerDIV", "abcdefghijklmnopqrstuvwxyz 0123456789 ABCDEFGHIJKLMNOPQRSTUVWXYZ .,/!\"{}[]<>" ) ;
</script>
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-19205690635184641212019-02-11T00:40:00.000+01:002019-02-11T00:40:36.085+01:00How to use Google fonts (even in this blog)Google provides a huge <a href="https://fonts.google.com/">list of fonts</a> and it describes <a href="https://developers.google.com/fonts/docs/getting_started">how to use them in detail</a>, in particular applying font effects and more.
<P>
I thought it worthwhile to capture the essence of using fonts and also I wanted to see if I could use Google fonts in this blog.
<P>
Using Google fonts boils down to three things actually:
<UL>
<LI><B>load</B> the font from Google by supplying a link in the <B><head></B> section of your HTML document
<PRE>
<link href="https://fonts.googleapis.com/css?family=Amarante" rel="stylesheet">
</PRE>
<LI>defining a corresponding <B>style</B> e.g. a <I>class</I> style with a name
<PRE>
<style>
.coolFont { font-family: Amarante }
</style>
</PRE>
<LI> <B>applying</B> this font to any element with
<PRE>
< ... class="coolFont" ... >
</PRE>
</UL>
<P>
Here is a <B>Javascript</B> code snippet (which I actually included in the HTML text) which dynamically loads <B>three</B> fonts and defines three correponding styles. It applies each of the styles to an <B>h1</B> element.
<PRE>
<script>
var fonts = [ 'Acme', 'Amarante', 'Audiowide' ]
for( i=0; i<fonts.length; i++ ) {
var style = document.createElement( 'STYLE' );
// Define class style '.font0', '.font1' etc.
style.innerHTML = ".font" + i + "{ font-family: " + fonts[i] + ";}"
document.getElementsByTagName('head')[0].appendChild( style );
var link = document.createElement( 'LINK' );
link.setAttribute( 'href', 'https://fonts.googleapis.com/css?family=' + encodeURI( fonts[i] ) );
link.setAttribute( 'rel', "stylesheet" ) ;
document.getElementsByTagName('head')[0].appendChild( link );
}
</script>
<h1 class="font0">Cool font, dude! </h1>
<h1 class="font1">Cool font, dude! </h1>
<h1 class="font2">Cool font, dude! </h1>
</PRE>
This will create a links like this
<PRE>
<link href="https://fonts.googleapis.com/css?family=Amarante" rel="stylesheet">
</PRE>
and styles
<PRE>
.font1 {
font-family: Amarante;
}
</PRE>
<script>
var fonts = [ 'Acme', 'Amarante', 'Audiowide' ]
for( i=0; i<fonts.length; i++ ) {
var style = document.createElement( 'STYLE' );
// Define class style '.font0', '.font1' etc.
style.innerHTML = ".font" + i + "{ font-family: " + fonts[i] + ";}"
document.getElementsByTagName('head')[0].appendChild( style );
var link = document.createElement('LINK' );
link.setAttribute( 'href', 'https://fonts.googleapis.com/css?family=' + encodeURI( fonts[i] ) );
link.setAttribute( 'rel', "stylesheet" ) ;
document.getElementsByTagName('head')[0].appendChild( link );
}
</script>
<P>
and here is the result
<P>
<h1 class="font0">Cool font, dude! </h1>
<h1 class="font1">Cool font, dude! </h1>
<h1 class="font2">Cool font, dude! </h1>
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-71465762260129622522019-01-24T15:25:00.000+01:002019-01-24T15:27:24.456+01:00How to get coloured output from 'find'When I need to get <b>coloured output</b> from shell scripts (e.g. in error messages) I am using the oldstyle escape sequences.
<P>
Example:
<br />
<pre>printf "\e[31mERROR - some error description\e[0m\n"
</pre>
which will result in
<br />
<pre><span style="color: red;">ERROR - some error description</span>
</pre>
<b>\e[31m</b> is basically ESCAPE and a colour code (31 for red, see <a href="https://en.wikipedia.org/wiki/ANSI_escape_code">ANSI escape codes</a>) for colour terminals. <b>\e[0m</b> ends the colouring.
<p />
I wanted to use the colour escape sequences in the <b>printf of find</b>.
<p />
This does <b>not</b> work when using <b>\e</b> and an error is thrown:
<pre>find . -printf "\e[31mSome Output\n" -quit
find: warning: unrecognized escape `\e'
\e[31mSome Output
</pre>
The solution is to <b>replace \e by its octal representation \033</b> (see e.g. <a href="http://www.asciitable.com/">ASCII table</a>).
<br />
<pre>find . -printf "\033[31mSome Output\n" -quit
<span style="color: red;">Some Output</span>
</pre>
<P>
Of course the <I>find</I> command in the example is not really useful (it quits right away) and it is only used to showcase the issue.
Here is a more useful invocation:
<P>
find empty and non-empty sub directories whereas non-empty should be coloured.
<PRE>
# Create example directories
mkdir -p tmp/a tmp/b tmp/c; touch tmp/b/foo
# Find
find tmp -type d \( -empty -printf "%-10p empty\n" \) -o \
\( ! -empty -printf "\033[31m%-10p not empty\033[0m\n" \)
<span style="color: red;">tmp not empty</span>
tmp/a empty
<span style="color: red;">tmp/b not empty</span>
tmp/c empty
</PRE>
i.e. I can distuingish in <i>find</i> between different cases and apply different colours according to my needs.Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-58816312391999054172018-12-10T15:42:00.000+01:002018-12-10T16:58:56.856+01:00How to write better/safer functions for UNIX commands through the example of 'mkdir'UNIX shell scripts contain loads of UNIX commands. Very often these commands are invoked in a simple manner which do not quite meet the standards of being <i>production worthy</i>.<br />
I don't want to discuss here what standards of production worthiness are but I want to show a few steps to improve your scripts from using a very simple command invocation to a more refined usage of the command.
<br />
My main consideration is this:
<br />
<ul>
<li><b>easiness of trouble shooting</b>: scripts will fail. The script should behave in a way to make it easy for a troubleshooter to identify the root cause of the failure.
</li>
</ul>
<br />
This implies:
<b>errors should be handled properly</b> i.e. they should be
<ul>
<li>caught,
</li>
<li>documented e.g. in log files and
</li>
<li>finished. Finishing means e.g. revoking certain actions being done beforehand, cleaning up temporary stuff, continuing with other actions or exiting the script (if appropriate).
</li>
</ul>
Before I start with my example I'd like to highlight an important thing which is also a main driver for this tutorial: many UNIX commands fail in a very simple manner. There can be a number of reasons for failure but the command always <b>exits with exit code 1</b> and writes a more or less descriptive <b>error message (to stderr)</b>. Not all of the failure reasons might lead to the same behaviour, reason A might be grave and imply exiting the script, reason B might be skipped.<br />
<br />
I'll explain this through the example of <code>mkdir</code>. As a teaser here is a common scenario:<BR>
scripts often create temporary directories at the start. If the temporary directory already exists you probably don't care and despite of a mkdir failure would like to continue. If mkdir fails because the script is executed in a wrong directory where you don't have write permissions you probably don't want to continue and rather exit. This should be handled in your code.
<P>
Here is the starting point: a simple invocation of <code>mkdir</code>.<br />
<code>mkdir tmpdir</code>
<br />
If this command fails the script will show an error message and probably exit e.g. if <code>set -e</code> is set in <i>bash</i>.
<br />
If you want to <b>catch</b> the error and decide what to do there are some options. You could use the shell binary operators AND (&&) or OR (||).
<br />
<ul>
<li>you definitely want to exit: <code>mkdir tmpdir || exit 1</code>
</li>
<li>you do not want to exit the script but write a specific message: <br /><code>mkdir tmpdir || echo "WARNING - mkdir of tmpdir failed" </code>
</li>
<li>you want to define some actions (which might include an exit): <pre>mkdir tmpdir
if [ $? -ne 0 ] ; then
# Do something here e.g.
echo "ERROR - mkdir of tmpdir failed. Exiting." >&2
exit 1
fi
</pre>
</li>
</ul>
Now assume that sometimes you want to exit if mkdir fails and sometimes not. An idea is to write a <b>wrapper function</b> which contains all the logic.
<br />
<pre># Function 'mkDir'
# $1: Y/N flag to exit (Y) or not (N) in case of failure
# $2: the directory to be created
function mkDir() {
mkdir "$2" # execute mkdir command
rc=$? # capture return code
# If successful return immediately
[ $rc -eq 0 ] && return 0
# Check if exit is required
if [ "$1" = "Y" -o "$1" = "y" ] ; then
echo "ERROR - mkdir failed for $2" >&2
exit 1
fi
echo "WARNING - mkdir failed for $2" >&2
return $rc
}
# invocation
mkDir Y tmpdir
# A loop to create a number of directories.
# We want to loop through all directories
# regardless if a mkdir fails or not and
# create a tmpfile in each of them.
for tmpdir in a b c ; do
mkDir n $tmpdir || continue
touch $tmpdir/tmpfile
done
</pre>
A trickier handling would be to <b>distinguish different errors</b>. Two main reasons for a mkdir failure are
<br />
<ul>
<li> the directory already exists
</li>
<li> the directory cannot be created because of issues with parent directory of the new directory (which could be <B>.</B> or an absolute path) like missing write permissions (<b>w</b> missing for owner, group or other, depending on who runs the script where)
</li>
</ul>
<br />
So here is an extended version of our wrapper function, also adding different <i>return codes</i> for various findings.
<BR>
<pre># Function 'mkDir'
# $1: Y/N flag to exit (Y) or not (N) in case of failure
# $2: the directory to be created
# Return codes:
# 0: all ok, directory could be created
# 1: mkdir failed (for a different reason than the assertions)
# 2: directory already exists
function mkDir() {
DIR="$2"
# Before executing mkdir we run a couple of assertions.
# Check if directory already exists
[ -d "$DIR" ] && echo "WARNING - directory $DIR already exists" && return 2
# Check if parent directory exists and is writable
# If not: we exit here
PARENTDIR=`dirname $DIR`
[ ! -d "$PARENTDIR" ] && echo "ERROR - parent directory $PARENTDIR does not exist" && exit 1
[ ! -w "$PARENTDIR" ] && echo "ERROR - parent directory $PARENTDIR is not writable" && exit 1
# Now we try to run the mkdir command
mkdir "$DIR"
rc=$? # capture return code
# If successful return immediately
[ $rc -eq 0 ] && return 0
# Here we know: mkdir failed but not for the two reasons we checked earlier
# Check if exit is required
if [ "$1" = "Y" -o "$1" = "y" ] ; then
echo "ERROR - mkdir of $DIR failed" >&2
exit 1
fi
echo "WARNING - mkdir of $DIR failed" >&2
return $rc
}
# Invocations
mkDir n tmpdir
mkDir n tmpdir # this one fails because tmpdir already exists
mkDir n abc/tmpdir # this one fails if abc does not exist
mkDir n /tmpdir # this one fails because we cannot write to a root owned directory
</pre>
<P>
The examples above were mainly showing how to <B>capture and finish</B> failures. I probably will write about <B>documenting errors</b> in a separate blog which will cover separation or unification of <I>stderr</I> and <I>stdout</I>, redirecting messages to a proper log file and if needed show messages in parallel on the terminal.
<h3> Conclusion</h3>
In order to make life easier when trouble shooting script failures it it advisable to <B>wrap UNIX commands</B> into functions and create some error handling logic for known and common error causes.
In the long run and in particular if used by a number of different persons you will save time since the script will throw out a specific defined error message. The various error causes of a command can drive different behaviour in the script which does help write better logic flows.
<h3>Note 1</h3>
Using <B>other programming languages</B> does not necessarily help. The Java <code>java.io</code> or <code>java.nio</code> packages would throw a Java <code>IOException</code>, again not very distinguishable and too simple.
<h3>Note 2</h3>
I wonder why there is no <B>shell library</B> around to cover this topic, basically a shell library of <B>UNIX command wrapper functions</B>. Maybe I have not searched enough.Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-85274901350223437152018-12-06T23:11:00.000+01:002018-12-10T17:00:48.588+01:00Thoughts about error messages and troubleshootingIn this article I will discuss a couple of thoughts about <b>error messages</b> and their helpfulness to <b>troubleshoot the real issue</b> or - as it turns out - often <B>inadequacy</B> in quickly troubleshooting errors and I also give an example at the very end how to do better.
<P>
My example is: <b>remove a directory</b> (in UNIX)
<br />
The usual command is something like <br />
<code>
rm -r somedir
</code>
<br />
Things can go wrong and you will end up with a <br />
<code>
rm: somedir: Permission denied
</code>
<P>
I bet many of the readers will have encountered this in their UNIX lives (sysadmin or not).<br />
The unfortunate thing for <b>troubleshooting</b> this error is that there can be <b>multiple causes</b> for it.
Depending on your experience you can more or less quickly spot the root cause in your real world scenario. The not so obvious fact - at least to the unexperienced user - is that in order to remove a directory you need sufficient permissions for the directory <i>and</i> also sufficent permissions for its parent directory.<br />
In more abstract terms going beyond <i>directory removal</i>: any action failing with <b>permission denied</b> probably requires permissions <B>(1)</B> for the <i>object to be handled</i> but <B>(2)</B> also for one or more <i>contextual objects</i>.
<br />
Going back to the <i>directory removal</i> here are number of permission issues which are all negatives of the necessary rule: <i>the user executing the <b>rm</b> needs to have <b>read</b> and <b>write</b> permissions on the directory and its parent directory (which requires <b>execute</b> permission too). </i> This can be achieved via user, group or other permission and ownership settings. <br />
(Note: I am not discussing even more refined settings via ACLs which is adding to complexity).
<br />
Even if both directory and parent directory are owned by the executing user there are number of error causes.
<br />
<ul>
<li> directory does not have write permissions
</li>
<li> directory has write permissions but no read permission
</li>
<li> parent directory does not have write or execute permissions
</li>
</ul>
Now after this long explanation I am coming to my actual message: <b>Error messages are often not precise enough to help in <underline>quick</underline> trouble shooting</b>. You need some meta knowledge and experience to identify the <b>root cause</b> which will allow you to <b>resolve</b> the issue.<br />
In the example above the standard UNIX <B>rm</B> is mapping a list of different root causes into a single <i>permission denied</i> error message leaving it to the user to know and investigate the various causes.<br />
I tried a different approach using <b>Java</b> and the Java <i>java.io.File</i> and <i>java.nio.file.Files</i> packages to delete a directory but their exceptions (<i>IOExcepction</i>) are equally simplistic and do not help to identify the root cause quickly.
<P>
<b>I am wondering why this is the case</b>, in general, not just reduced to this example.<br />
My experience in <b>Application Support</b> is that it is most important to quickly identify root causes. Why should a user be left with poking around rather than giving him/her the actual root cause right away?
<br />
If the <b>rm</b> command knows that the write permission for the directory is missing why doesn't it tell us right away?
<br />
I am seeing this way of thinking at all kinds of levels in Software development. Rather than giving all the information about an error scenario the user is stuck with some general blurb and needs to spend time. My example is from a rather low level and things get more frustrating the further one moves up the software ladder. Complex GUI based applications are even more frustrating to troubleshoot.
<br />
<hr />
Here is a script to generate some directories with wrong permissions and test <i>rm</i>. I am creating 3 directories with different permissions. Two removals will fail <br />
<pre>
mkdir -m 000 somedir1
mkdir -m 200 somedir2
mkdir -m 600 somedir3
ls -la
rm -r somedir1 somedir2 somedir3
ls -la
</pre>
<br />
Here is the output:
<pre>
total 0
drwxr-x--- 5 ahaupt 2006807681 170 Dec 6 16:51 .
drwxr-xr-x 10 ahaupt 2006807681 340 Dec 6 13:46 ..
d--------- 2 ahaupt 2006807681 68 Dec 6 16:51 somedir1
d-w------- 2 ahaupt 2006807681 68 Dec 6 16:51 somedir2
drw------- 2 ahaupt 2006807681 68 Dec 6 16:51 somedir3
rm: somedir1: Permission denied
rm: somedir2: Permission denied
drwxr-x--- 4 ahaupt 2006807681 136 Dec 6 16:51 .
drwxr-xr-x 10 ahaupt 2006807681 340 Dec 6 13:46 ..
d--------- 2 ahaupt 2006807681 68 Dec 6 16:51 somedir1
d-w------- 2 ahaupt 2006807681 68 Dec 6 16:51 somedir2
</pre>
Now let's re-create somedir3 again but remove the write permission for the current directory. All 3 removals will fail.<br />
<pre>
mkdir -m 600 somedir3
chmod u-w .
ls -la
rm -r somedir1 somedir2 somedir3
ls -la
</pre>
<br />
<pre>
dr-xr-x--- 5 ahaupt 2006807681 170 Dec 6 16:55 .
drwxr-xr-x 10 ahaupt 2006807681 340 Dec 6 13:46 ..
d--------- 2 ahaupt 2006807681 68 Dec 6 16:51 somedir1
d-w------- 2 ahaupt 2006807681 68 Dec 6 16:51 somedir2
drw------- 2 ahaupt 2006807681 68 Dec 6 16:55 somedir3
rm: somedir1: Permission denied
rm: somedir2: Permission denied
rm: somedir3: Permission denied
total 0
dr-xr-x--- 5 ahaupt 2006807681 170 Dec 6 16:55 .
drwxr-xr-x 10 ahaupt 2006807681 340 Dec 6 13:46 ..
d--------- 2 ahaupt 2006807681 68 Dec 6 16:51 somedir1
d-w------- 2 ahaupt 2006807681 68 Dec 6 16:51 somedir2
drw------- 2 ahaupt 2006807681 68 Dec 6 16:55 somedir3
</pre>
<P>
<hr>
Finally here is a *shell snippet* which gives proper information by catching some of the root causes of a failing <B>rm</B>. This is not exhaustive but should give you an idea what could be done to improve the error handling and make it more user friendly.
<PRE>
function rmDir {
DIR="$1"
PARENTDIR=`dirname $1`
MSG=""
# Check write permission of dir
[ ! -w "$DIR" ] && MSG="$MSG
ERROR - not writable $DIR"
# Check read permission of dir
[ ! -r "$DIR" ] && MSG="$MSG
ERROR - not readable $DIR"
# Check write permission of parent dir
[ ! -w "$PARENTDIR" ] && MSG="$MSG
ERROR - not writable $PARENTDIR"
# Check execute permission of parent dir
[ ! -x "$PARENTDIR" ] && MSG="$MSG
ERROR - not executable $PARENTDIR"
[ ! -z "$MSG" ] && echo "$MSG" && exit 1
rm -r $DIR
}
</PRE>
In a scenario like below you will get adequate error messages and there is not even a try to invoke the real <B>rm</B>.
<PRE>
dr-xr-x--- 4 ahaupt 2006807681 136 Dec 6 22:52 .
drwxr-xr-x 11 ahaupt 2006807681 374 Dec 6 22:57 ..
d-w------- 2 ahaupt 2006807681 68 Dec 6 16:51 somedir2
rmDir somedir2
ERROR - not readable somedir2
ERROR - not writable .
</PRE>
<hr>
<P>
Of course it is difficult to think about all possible error cases in advance but I claim that there are many cases where it would be possible to enhance the code and show more detailed and more precise messages than a simple mapping of many kinds of errors into a vague error statement.Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-1668101089041336762017-04-18T11:44:00.001+02:002017-04-18T11:54:06.171+02:00Create bootable USB stick with Linux for old Macbook with MacOS 10.5.8Since my old MacBook with MacOS 10.5.8 is only running with outdated browser versions I am having more and more troubles looking at <b>modern</b> websites which are using fancy features (LinkedIn, Ryanair, ...). Since I cannot upgrade the machine I needed a workaround to access up-to-date browsers. An idea was to <B>create a bootable USB stick with Linux</B>.
<br />
You basically need to
<li>download a Linux image <b>iso</b> file
</li>
<li>prepare the USB stick so that it can receive the iso file
</li>
<li>copy the iso file onto the stick</li>
<P>
There are GUI tools out there which can achieve this but I prefer the simple <b>command line</b> set of instructions which I could not find for MacBooks. I did find instructions for creating a USB stick on a Linux or Windows box though which I combined with instructions for the particular device handling on MacBooks.
<BR>
Main resources:
<li><a href="http://askubuntu.com/questions/372607/how-to-create-a-bootable-ubuntu-usb-flash-drive-from-terminal">http://askubuntu.com/questions/372607/how-to-create-a-bootable-ubuntu-usb-flash-drive-from-terminal</a></LI>
<P/>
<h2>
Download Linux image</h2>
Download your favourite Linux image which you want to put onto the USB stick.
I opted for <b>Lubuntu</b> from <a href="https://help.ubuntu.com/community/Lubuntu/GetLubuntu">https://help.ubuntu.com/community/Lubuntu/GetLubuntu</a><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsDZc3Ptp0oxMDZueLAQRy2oZfpGuVcSYBW0a8FrWdqZSb2jegq9gDQNm8jZ46qWyxMCYHAp9NLffxViBr4NvtusmcQ15k5BYZRsXo8qyEFA4-MgsnoeNscfoLzBSX8MaeTvICtk6L-p8/s1600/Lubuntu_GetLubuntu.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="91" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsDZc3Ptp0oxMDZueLAQRy2oZfpGuVcSYBW0a8FrWdqZSb2jegq9gDQNm8jZ46qWyxMCYHAp9NLffxViBr4NvtusmcQ15k5BYZRsXo8qyEFA4-MgsnoeNscfoLzBSX8MaeTvICtk6L-p8/s320/Lubuntu_GetLubuntu.png" width="320" /></a></div>
where I chose the <b>PC 64bit Standard image disc</b>.
<p />
<h2>
Attach the USB stick</h2>
Attach the USB stick to a USB port. It will be shown in <b>Finder</b>. Check on the system:<BR>
<b>mount points</b>: my USB stick is mounted under <b>disk2</b>, to be precise: the file system in partition <b>s1</b> is mounted.<br>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2lqy4zy1odcunpx3wlmSLeb2DZkMhUlzwYOPO1_pq2VrXAkdmdBdN8osrGBrXRkOhoRJvGu34FK0XojQFPsDP1wDw41mzZugCkE0RH1o8eT1Wj_Fm6AAAH_CgzZsW5zWTtq-8ZAZ_guc/s1600/mount.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="116" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2lqy4zy1odcunpx3wlmSLeb2DZkMhUlzwYOPO1_pq2VrXAkdmdBdN8osrGBrXRkOhoRJvGu34FK0XojQFPsDP1wDw41mzZugCkE0RH1o8eT1Wj_Fm6AAAH_CgzZsW5zWTtq-8ZAZ_guc/s1600/mount.jpg" width="640" /></a></div>
<br />
<b>ls</b> of <b>disk</b> devices: <b>disk2</b> is the whole device, <b>disk2s1</b> is the partition with the file system
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUvssg_iKejhcW0Fx7DCcWjcw-2SPtKu2MDtBd7lygeUiym1vsDBY3ahQ815z2csHne6tjNvFkuRW3AdppG_wR5aeslrASDlr_uIo8ThxKktQkZlMFHjBhA75fmMy4FudDcjKLffMxcJQ/s1600/ls+dev+disk.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="176" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUvssg_iKejhcW0Fx7DCcWjcw-2SPtKu2MDtBd7lygeUiym1vsDBY3ahQ815z2csHne6tjNvFkuRW3AdppG_wR5aeslrASDlr_uIo8ThxKktQkZlMFHjBhA75fmMy4FudDcjKLffMxcJQ/s1600/ls+dev+disk.jpg" width="640" /></a></div>
<p />
<h2>
Unmount the partition</h2>
What we want is: keep the USB stick attached and in the list of devices but unmount all partitions. <b>Eject</b> in <b>Finder</b> is no option since it will completely remove the device. We only want to <b>unmount</b> the USB stick.Overall this was the trickiest part since the normal <b>UNIX umount</b> did not work as expected.
The usual <b>umount</b> does not work on MacOS since there are applications (spotlight, etc.) constantly accessing the file system so an error is thrown (of course this command and all the following need to be done as <b>root</b> therefore I am using <b>sudo</b>):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1zooVb679H4qo45gLVj0tnNzsZMxoVEK7eMncx6_K9Cfpq2VlXUuPLYg6AEvJTa81j_SjVHbWc9vUNCULk42JzbllNCvntNxoL_3rU7NosCaVzq8dnJYQ2m-B9UIzSVAXt6IOsflVXls/s1600/failed+unmount.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="32" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1zooVb679H4qo45gLVj0tnNzsZMxoVEK7eMncx6_K9Cfpq2VlXUuPLYg6AEvJTa81j_SjVHbWc9vUNCULk42JzbllNCvntNxoL_3rU7NosCaVzq8dnJYQ2m-B9UIzSVAXt6IOsflVXls/s1600/failed+unmount.jpg" width="400" /></a></div>
<br />
One could try to kill all these processes (after first identifying them) but the real trick is to use the Mac specific <b>diskutil</b> command:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKi5J-4-IB1HQqfRq9N-8b_IBV9HGQOKiK0N_1tHfytRxoGigmldu6rXbKfK0H82OoiMHs7hvXqR690IdA-xo2njkOhc1e5RVKDznSBNskAknahRcOn0q294Etco5f9czJGc-VJ2_r14s/s1600/unmount.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="55" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKi5J-4-IB1HQqfRq9N-8b_IBV9HGQOKiK0N_1tHfytRxoGigmldu6rXbKfK0H82OoiMHs7hvXqR690IdA-xo2njkOhc1e5RVKDznSBNskAknahRcOn0q294Etco5f9czJGc-VJ2_r14s/s1600/unmount.jpg" width="400" /></a></div>
<br />
<h2>
Write Linux image to USB stick</h2>
The good old <b>dd</b> is all you need. Input file <b>if</b> is the Linux iso file, output file <b>of</b> is the USB device (note the full device <b>/dev/disk2</b>, not a particular partition), and block size 4MB (could be anything which you think feasible on your machine).<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgenIgzLR-Fc1D4bK-INfExDs7M4uZnaoB8__WtGtCfdLtvP6eJ0aGF_W_F15fSq-4TrnbgRCb_8jhsW-uu68Hsb22eUxpQi3WZVjAFiWVkJy55nU0MJfVOt-sZj1t4LmCsBtMtooX3ijU/s1600/dd+to+USB.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="72" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgenIgzLR-Fc1D4bK-INfExDs7M4uZnaoB8__WtGtCfdLtvP6eJ0aGF_W_F15fSq-4TrnbgRCb_8jhsW-uu68Hsb22eUxpQi3WZVjAFiWVkJy55nU0MJfVOt-sZj1t4LmCsBtMtooX3ijU/s1600/dd+to+USB.jpg" width="640" /></a></div>
<br />
<B>Note</B>: this takes some time, 22 minutes in my case.
<p />
<h2>Reboot and choose USB stick</H2>
Now when rebooting the machine and pressing the <b>Option</b> button after the sound there is screen showing the old Mac harddisk and an EFI as boot options.
<br />
<br />
</li>
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-19044358430137195412015-04-23T22:22:00.000+02:002015-04-23T22:29:35.648+02:00A hack to continue with Skype 6.3 on MacOS 10.5.8Today Skype users on MacOS 10.5.8 were surprised again that Skype rejected the login and requested to upgrade to a new version. Navigating to the new version download announced once again:
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLxAc881L2sqWQHXoy4md2T93EI0PGjdrg8zAAxPjmJfe1HKLgM_V5hQdTLf7d7zT6_nKiF8Bw2RzGdnWKNG7A4x-8umTQldeUIrULaSI_bCVHTiWGZLg1KeainCjCyPclLlXQghkCjIM/s1600/NoSkypeForMacOS10.5.tiff" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLxAc881L2sqWQHXoy4md2T93EI0PGjdrg8zAAxPjmJfe1HKLgM_V5hQdTLf7d7zT6_nKiF8Bw2RzGdnWKNG7A4x-8umTQldeUIrULaSI_bCVHTiWGZLg1KeainCjCyPclLlXQghkCjIM/s640/NoSkypeForMacOS10.5.tiff" /></a></div>
<P>
This had happened in the past and the solutions then ranged from installing every old versions of Skype (2.8) to tampering with certain system settings, last year Skype released an officially supported version 6.3 for MacOS 10.5.8 but this only lasted until today.
<P>
So I tried various things and a variation of earlier solutions did it for me which I would like to share with the community. In summary: trick Skype into thinking that you are running a higher version than 6.3.
<h3>Change 3 settings in the info.plist file</h3>
<P>
<LI>First of all quit Skype so that no Skype process is running anymore
<LI>Open info.plist with the property editor (in <b>Finder -> Applications</b> choose <b>Skype</b>, right-mouse click and choose <b>Show Package Contents</b>, in the new window double-click on <b>Contents</b> and double-click on <b>info.plist</b> )
<LI>Change
<PRE>
<TABLE>
<TR><TD>BuildMachineOSBuild</TD><TD> 12F45</TD></TR>
<TR><TD>Bundle versions string, short</TD><TD> 6.15</TD></TR>
<TR><TD>Bundle version</TD><TD> 6.15.0.330</TD></TR>
</TABLE>
</PRE>
The previous settings were for me <CODE>6.3.0.604</CODE> and <CODE>12C3103</CODE>.
<LI>Close the property editor and <b>Save</b> the file
<P>
<h3>Change the /etc/hosts file</h3>
<P>
This prevents that your network contacts the Skype download server.
<LI>Edit the <CODE>/etc/hosts</CODE> file (with your editor of choice)
<LI>Enter this line <PRE>127.0.0.1 ui.skype.com</PRE>
<LI>Start Skype and login again
<P>
There you go.<BR>
Let's wait and see until this solution becomes obsolete.Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com12tag:blogger.com,1999:blog-7194384172575329109.post-33035886199431340532013-08-07T00:41:00.000+02:002013-08-07T00:41:12.420+02:00Creating a MySQL db on Ubuntu as a normal userLately I tried to create a <b>MySQL</b> db on <b>Ubuntu</b> (version 11 which has MySQL 5.1 preinstalled).
I was logged in under my normal username but I got a surprise when running the <b>mysql_install_db</b> command.
<BR>
<PRE>
$ <b>/usr/bin/mysql_install_db --datadir=./mysql/data</b>
Installing MySQL system tables...
130806 22:17:21 [Warning] Can't create test file /home/andreash/mysql/data/andreas-Ub-2.lower-test
130806 22:17:21 [Warning] Can't create test file /home/andreash/mysql/data/andreas-Ub-2.lower-test
Installation of system tables failed! Examine the logs in
./mysql/data for more information.
...
</PRE>
<P>
There were not log files though and checking directories and permissions didn't reveal any problems.
<BR>So I started to search and found that <b>Ubuntu</b> uses a security mechanism called <a href="https://wiki.ubuntu.com/AppArmor">apparmor</a> which can be used to control certain aspects of an application.<BR>
In regards to MySQL that means that there exists a MySQL profile which defines which directories can be accessed (and how) by the MySQL programs.
The profile for the daemon <b>mysqld</b> is defined in <b>/etc/apparmor.d/usr.sbin.mysqld</b> and looks like this:
<PRE>
# Last Modified: Tue Jun 19 17:37:30 2007
#include <tunables/global>
/usr/sbin/mysqld {
#include <abstractions/base>
#include <abstractions/nameservice>
#include <abstractions/user-tmp>
#include <abstractions/mysql>
#include <abstractions/winbind>
capability dac_override,
capability sys_resource,
capability setgid,
capability setuid,
network tcp,
/etc/hosts.allow r,
/etc/hosts.deny r,
/etc/mysql/*.pem r,
/etc/mysql/conf.d/ r,
/etc/mysql/conf.d/* r,
/etc/mysql/*.cnf r,
/usr/lib/mysql/plugin/ r,
/usr/lib/mysql/plugin/*.so* mr,
/usr/sbin/mysqld mr,
/usr/share/mysql/** r,
/var/log/mysql.log rw,
/var/log/mysql.err rw,
/var/lib/mysql/ r,
/var/lib/mysql/** rwk,
/var/log/mysql/ r,
/var/log/mysql/* rw,
/{,var/}run/mysqld/mysqld.pid w,
/{,var/}run/mysqld/mysqld.sock w,
/sys/devices/system/cpu/ r,
# Site-specific additions and overrides. See local/README for details.
#include <local/usr.sbin.mysqld>
}
</PRE>
<P>
So in order to enable MySQL to access a subdirectory of my $HOME I had to edit the file as root (sudo vi ...) and add this line to the list (I put it right under the /sys/devices line)
<PRE>
/home/andreas/mysql/** rw,
</PRE>
<BR>
The apparmor man page explains the syntax and attributes in detail.
For my purposes it suffices to know that <b>**</b> stands for the directory and all subdirectories underneath and <b>rw</b> is of course read/write.
<P>
Then this new profile needs to be activated replacing the old one via
<PRE>
$ <b>sudo apparmor_parser -rv /etc/apparmor.d/usr.sbin.mysqld</b>
Replacement succeeded for "/usr/sbin/mysqld".
</PRE>
<P>
Finally running the MySQL program again did create the databases.
<PRE>
Installing MySQL system tables...
OK
Filling help tables...
OK
To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system
PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:
...
</PRE>
<HR>
Not knowing much about apparmor yet I wonder how one would go about to allow all users (on a bigger multi-user server) to use MySQL or any other application which is secured in the same way.
It would be impractical to add all users home directories to the profile file so I guess there must be some shortcut. This needs more reading.Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-42942278476167791412013-08-02T19:09:00.001+02:002013-08-02T19:09:45.507+02:00LibreOffice calc: automatically sort data upon data entryA common task in LibreOffice (or Apache OpenOffice) <b>calc</b> is to sort a range of cells via <b>Data -> Sort...</b> where the sort criteria (columns, sort order) are defined.
<BR>
If the spreadsheet is a living document where entries are deleted and new entries entered on a regular basis one would wish for this sorting to be done automatically rather than the user having to do it manually each time.
In this article I show how to achieve this with <b>macros</b> in LibreOffice.
<P>
The very basic idea of automation is to <b>assign a macro to an event</b>. In case that event happens the macro is being run.
<P>
But before I get to that I'd like to outline the steps and how they are connected.
<LI>First define a range for the cells to be sorted
<LI>Then create the macro which will be triggered (it will make use of the range)
<LI>Assign the macro to the 'change content' event
Now whenever the content of a cell is changed the macro will be triggered and run.
<P>
<H2>Define a range</H2>
Assume the data to be sorted are in two columns A and B. Currently I have only 4 entries but there could be more in the future. The columns also have a header in the first row.
<BR>
The range is set by
<LI>marking columns A and B
<LI>going to <b>Data -> Define Range...</b> and entering a name for the selection and finally clicking <b>Add</b> and <b>OK</b>. I chose the name 'MyData'.
<BR><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5SG1HQMu77KzNQW87cp0A3Ymtj6KGlxW5WmJFDRZNFOQCw2v1MxkSt0yt11h4GensPJVyeIEQsuKiW0IXCN0bJO_ec6AnPM2HiG2-q7K3OfIvJQtz-6ICin550w9QegrmpyMlwK84FhM/s1600/OOO_Define_Range.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5SG1HQMu77KzNQW87cp0A3Ymtj6KGlxW5WmJFDRZNFOQCw2v1MxkSt0yt11h4GensPJVyeIEQsuKiW0IXCN0bJO_ec6AnPM2HiG2-q7K3OfIvJQtz-6ICin550w9QegrmpyMlwK84FhM/s400/OOO_Define_Range.tiff" /></a>
<P>
<H2>Create a macro</H2>
It would suffice to create one macro but I opted to split the macro into two, one being of a more general reusable nature and the other more specific to the event.
<P>
The first macro 'SortRange' sorts a range by its second column in descending order using the first row as header i.e. not part of the actual sort. The macro gets a range object as parameter and thus could be used for any range.<P>
<PRE>
<b>Sub SortRange( oRange As Variant ) </b>
rem Sorts range 'sRange' by column 2 in descending order
rem (assumes that columns have labels in first row)
rem ----------------------------------------------------------------------
rem define variables
dim document as object
dim dispatcher as object
rem ----------------------------------------------------------------------
rem get access to the document
document = ThisComponent.CurrentController.Frame
dispatcher = createUnoService("com.sun.star.frame.DispatchHelper")
rem ----------------------------------------------------------------------
rem Select the range
dim args1(0) as new com.sun.star.beans.PropertyValue
args1(0).Name = "ToPoint"
args1(0).Value = oRange.AbsoluteName
dispatcher.executeDispatch(document, ".uno:GoToCell", "", 0, args1())
rem ----------------------------------------------------------------------
dim args2(7) as new com.sun.star.beans.PropertyValue
args2(0).Name = "ByRows"
args2(0).Value = true
<b>args2(1).Name = "HasHeader"
args2(1).Value = true</b>
args2(2).Name = "CaseSensitive"
args2(2).Value = false
args2(3).Name = "NaturalSort"
args2(3).Value = false
args2(4).Name = "IncludeAttribs"
args2(4).Value = true
args2(5).Name = "UserDefIndex"
args2(5).Value = 0
Rem Sort by the second column of the range !!!
<b>args2(6).Name = "Col1"
args2(6).Value = oRange.RangeAddress.StartColumn + 2</b>
<b>args2(7).Name = "Ascending1"
args2(7).Value = false</b>
dispatcher.executeDispatch(document, ".uno:DataSort", "", 0, args2())
End Sub
</PRE>
Note how the <b>Data -> Sort...</b> <b>Options</b> are mapped to these properties and could be adjusted if one wanted different sorting criteria.
<P>
The second macro 'SortRangeFilter' checks whether the change actually happened inside the range and then calls the SortRange macro. The <b>name of the range is hardcoded </b>in this macro and thus makes it inflexible but I couldn't find a better way (yet).
<BR>
Macros which are triggered by an event listener get passed a parameter which describes the source object. These can be different things which I need to differentiate in my code:
<LI>a single cell e.g. when entering a value
<LI>a range of cells e.g. when deleting a row
<LI>a set of ranges e.g. when marking and deleting two disjunct cells
<BR>
In the last cases my selection might contain cells which are not members of my defined range.
In order to make my code the most flexible I apply the following logic:
<LI>a single cell is just a special case of range of cells thus I only need to differentiate two cases
<LI>for the remaining cases I determine the intersection of my named range with the selection (= source of the event). If it is empty there is nothing to do, otherwise I need to sort the range.
<P>
<PRE>
<b>Sub SortRangeFilter( oEvent As Variant )</b>
Dim sRange As String
sRange = "MyData"
Rem ^^^^^^
Rem Hardcoded range name
Rem Get the range object in order to access its boundaries
Dim oSheet As Variant
oSheet = ThisComponent.CurrentController.getActiveSheet()
Rem Define an error handler in case the range name does not exist
On Error Goto ErrorHandler
Dim oCellRange As Variant
oCellRange = oSheet.getCellRangeByName( sRange )
Rem Check the type of 'oEvent'
Rem Both support the 'queryIntersection' method
If ( oEvent.supportsService( "com.sun.star.sheet.SheetCellRanges" ) Or _
oEvent.supportsService( "com.sun.star.sheet.SheetCellRange" )) Then
If oEvent.queryIntersection( oCellRange.RangeAddress ).count > 0 Then
SortRange( oCellRange )
Endif
Endif
Exit Sub
ErrorHandler:
Msgbox "SortRangeFilter: Range does not exist '" + sRange + "'"
Exit Sub
End Sub
</PRE>
Note: rather than hardcoding the range name one could check all defined ranges for the current sheet and try to find the right one but there are a number of drawbacks:
<LI>a cell could be a member of several ranges (since ranges can overlap). Which range should be picked for sorting?
<LI>if multiple cells are selected (e.g. a row) there is an even higher chance of confusion. Some of the selected cells might be in range 1, others in range 2 etc. so which range is to be picked? And it is unlikely that all ranges need to be sorted.
<P>
Both macros need to be entered in a macro library. I chose <b>My Macros -> Standard -> Module1</b>.
<P>
<H2>Set event listener</H2>
Now that the macro is in place I can set up the event trigger.
<LI>Right click on the sheet name and choose <b>Sheet Events...</b>
<LI>In the Assign Action window select <b>Content changed</b> and click <b>Macro...</b>
<LI>In the Macro Selector window navigate to (in my case) <b>My Macros -> Standard -> Module1</b> and select <b>SortRange</b> in the list of macros and click <b>OK</b>
<BR><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiUrWqKtsFcKn_gYd-vM7AZzu-OejhVzEtxf1LmhNzCozF8QJw1dCsAGXmpZSzp4lX8exVqUWAVWA0OeRNcZzlaI6g_4LjfDqtRNJEf1a0tgilid4UQu4oxF1TPNY4Qi6vMRMkWovbrok/s1600/OOO_Assign_Action.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiUrWqKtsFcKn_gYd-vM7AZzu-OejhVzEtxf1LmhNzCozF8QJw1dCsAGXmpZSzp4lX8exVqUWAVWA0OeRNcZzlaI6g_4LjfDqtRNJEf1a0tgilid4UQu4oxF1TPNY4Qi6vMRMkWovbrok/s400/OOO_Assign_Action.tiff" /></a>
<LI>Back again in the Assign Action window click <b>OK</b> too
<P>
Now you have it: <b>any change in the range i.e. columns A or B will trigger a re-sort</b>. Any other change on the sheet won't.
<BR>
If you create the range in a different place the sort will still be triggered.
<P>
<HR>
<HR>
Below I'll show a couple of scenarios and how the macros behave. Assume there is a defined range 'MyData' somewhere on the spreadsheet, say C5:D10. The column headers have been entered in B5 and C5.
<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUqJWVElhgGMLcwNKHdAFO1jqkOfCjGnxtQ2XWxN0gxB9bYZx7SiM0mYdzzDqIBxw6JqmYjWUfeSc7Sig0pbeTQCPXgiunzGRcM6MEUlwxfCn1EjsJ6OU0ZQJ-9znWMb_7mDi52_H873Y/s1600/OOO_Select_Range_Init.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUqJWVElhgGMLcwNKHdAFO1jqkOfCjGnxtQ2XWxN0gxB9bYZx7SiM0mYdzzDqIBxw6JqmYjWUfeSc7Sig0pbeTQCPXgiunzGRcM6MEUlwxfCn1EjsJ6OU0ZQJ-9znWMb_7mDi52_H873Y/s320/OOO_Select_Range_Init.tiff" /></a>
<P>
Then the sheet event trigger needs to be assigned as described above and I can start entering the data.
<P>
<TABLE>
<TR>
<TD>
I begin with <b>B6=A and C6=20</b>. Nothing happens other than the named range being marked due to the macro.<P>
Then I enter <b>B7=B and C7=34</b> and now the sort trigger already fires and sorts my columns and reverses the order.
</TD><TD>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1S-rYbreeLiK2d_575Qwo_yG57P_miR8pKbCmQEGLPhyphenhyphen6fpaAGxXCSIYt_prcq4oFGMfa3LiQ8zwrlMKSCgFPkx_9pN55k6YHOi0jwJcVhn7e3wI-E-TDnRkL5cnfcxG-IlisMMQMICo/s1600/OOO_Select_Range_2nd_data.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1S-rYbreeLiK2d_575Qwo_yG57P_miR8pKbCmQEGLPhyphenhyphen6fpaAGxXCSIYt_prcq4oFGMfa3LiQ8zwrlMKSCgFPkx_9pN55k6YHOi0jwJcVhn7e3wI-E-TDnRkL5cnfcxG-IlisMMQMICo/s320/OOO_Select_Range_2nd_data.tiff" /></a>
</TD>
</TR><TR>
<TD>I enter another data set <b>B8=C and C8=51</b>
</TD><TD>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg37SeScfhK93rq1ZX2BoG6Ne9sE-qM9_Td191kRXhOaY-GTgzzPcMdxd6u4pPbwx9R7Ye9e8FF-COEThWq3BVV3w9LhA_klT0xgg-iVLEoC-BQtPxc8ahWqim_TtbitxVw2xfwZwIyDCU/s1600/OOO_Select_Range_3rd_data.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg37SeScfhK93rq1ZX2BoG6Ne9sE-qM9_Td191kRXhOaY-GTgzzPcMdxd6u4pPbwx9R7Ye9e8FF-COEThWq3BVV3w9LhA_klT0xgg-iVLEoC-BQtPxc8ahWqim_TtbitxVw2xfwZwIyDCU/s320/OOO_Select_Range_3rd_data.tiff" /></a>
</TD>
</TR><TR>
<TD>
And finally the last data <b>B9=D and C9=40</b> which is moved up two rows.
</TD><TD>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdfXeS3w50Cr4C7mMmPeQzPXKX8bcC-o5Ab5WXAgM1Ajx5Vzq-GpoGXWgJx-bS92qG5bzdG0JSRjuXLSikdKijg3nsGQ39WEsfLWHfdpd47dq0DWZZMqal_m0rgOzbAMA9yWZRs78BI-8/s1600/OOO_Select_Range_4th_data.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdfXeS3w50Cr4C7mMmPeQzPXKX8bcC-o5Ab5WXAgM1Ajx5Vzq-GpoGXWgJx-bS92qG5bzdG0JSRjuXLSikdKijg3nsGQ39WEsfLWHfdpd47dq0DWZZMqal_m0rgOzbAMA9yWZRs78BI-8/s320/OOO_Select_Range_4th_data.tiff" /></a>
</TD>
</TR>
</TABLE>
<P>
Always starting with the last setup I show various scenarios to <b>delete</b> cells and how the sort automatically kicks in.
<P>
<TABLE>
<TR>
<TH COLSPAN=2>
Mark a single cell and delete it
</TH>
</TR><TR>
<TD>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVY1pALjQ912Rg95iQxGSmt8R6a1N4XjAdxKGAlMiVOpV0r5V0pepRqPTnPHCSTR5fsxXFB2QMQ7gDRa0JkAY68uvTfjXmVrjPM4UI5T_2qEmai5y5PF1_0a1eb6Z2NNJYnJvKdpjxlD0/s1600/OOO_Delete_Single_1.tiff" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVY1pALjQ912Rg95iQxGSmt8R6a1N4XjAdxKGAlMiVOpV0r5V0pepRqPTnPHCSTR5fsxXFB2QMQ7gDRa0JkAY68uvTfjXmVrjPM4UI5T_2qEmai5y5PF1_0a1eb6Z2NNJYnJvKdpjxlD0/s320/OOO_Delete_Single_1.tiff" /></a>
</TD><TD>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiH2tNt1elTv9NKex15V9MHl7zhcSjMJ6g2BGI6U-97ZR-VPBP43aI4fArfTRfl4zrDbfQtiGmKtwQ_f_DvbJRjG7eipmsow-9zdN6BVhC6XTRlEDNCks87PLMSWdeoM_v8YsmnDxaXFck/s1600/OOO_Delete_Single_2.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiH2tNt1elTv9NKex15V9MHl7zhcSjMJ6g2BGI6U-97ZR-VPBP43aI4fArfTRfl4zrDbfQtiGmKtwQ_f_DvbJRjG7eipmsow-9zdN6BVhC6XTRlEDNCks87PLMSWdeoM_v8YsmnDxaXFck/s320/OOO_Delete_Single_2.tiff" /></a>
</TD>
</TR>
<TR>
<TH COLSPAN=2>
Mark a row and delete it
</TH>
</TR><TR>
<TD>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTrhbk7N8j3femKKZcRt4B6B_HlsSNZTkV5nk-6RnY2gtzzsFZ0AsdCgyiFALq1B0jwY6GaxozEH2TGrFnDUEmE73y7FlgOepgjFWdPZ7kIW_W8TcC13WT_hO074U66eRww4OtxiG489g/s1600/OOO_Delete_Range_1.tiff" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTrhbk7N8j3femKKZcRt4B6B_HlsSNZTkV5nk-6RnY2gtzzsFZ0AsdCgyiFALq1B0jwY6GaxozEH2TGrFnDUEmE73y7FlgOepgjFWdPZ7kIW_W8TcC13WT_hO074U66eRww4OtxiG489g/s320/OOO_Delete_Range_1.tiff" /></a>
</TD><TD>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh16DAhcmnfEmsSDerxgFGZojTYOTikfK-fCT-vMMHOiIapJQ9_rwLLD08_NbUfCA4auxY5DW6hB8DcwGz5zwEkdHz0C5SD7yTIT3ZOw0Rn8I4yOpXCnx0QPW3omXeLqu005Ru0L8iqr8U/s1600/OOO_Delete_Range_2.tiff" imageanchor="1" style="clear: left; float: right; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh16DAhcmnfEmsSDerxgFGZojTYOTikfK-fCT-vMMHOiIapJQ9_rwLLD08_NbUfCA4auxY5DW6hB8DcwGz5zwEkdHz0C5SD7yTIT3ZOw0Rn8I4yOpXCnx0QPW3omXeLqu005Ru0L8iqr8U/s320/OOO_Delete_Range_2.tiff" /></a></div>
</TD>
</TR>
<TR>
<TH COLSPAN=2>
Mark some disjunct cells and delete them
</TH>
</TR><TR>
<TD>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjId4cxUeTTo_XSTrV4YrHcuEzaozSu656fZKzdEj8Mn9svHMvBSGyqz0MLlZ1QNnAyfd8qzWg9_D8-0NiG1qxXrUqXJ0RVul1UhXjNJcTvA-jb7NwgPEhYmAsv7CTuqV_dIPn51JzR-uY/s1600/OOO_Delete_Ranges_1.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjId4cxUeTTo_XSTrV4YrHcuEzaozSu656fZKzdEj8Mn9svHMvBSGyqz0MLlZ1QNnAyfd8qzWg9_D8-0NiG1qxXrUqXJ0RVul1UhXjNJcTvA-jb7NwgPEhYmAsv7CTuqV_dIPn51JzR-uY/s320/OOO_Delete_Ranges_1.tiff" /></a>
</TD><TD>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixa3gpcN5l5tEWmY-MaCtW_nCbTbsglxiUO_RZugUemSy1_4HfG0MwFxNKYGEbsgVgdZedsWubp7x6ZICPpbO3gbES4dwvco8CZl_kZGMFy7_nWavb9-z4TcUhBzTUnVGlGhZaegdCLiM/s1600/OOO_Delete_Ranges_2.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixa3gpcN5l5tEWmY-MaCtW_nCbTbsglxiUO_RZugUemSy1_4HfG0MwFxNKYGEbsgVgdZedsWubp7x6ZICPpbO3gbES4dwvco8CZl_kZGMFy7_nWavb9-z4TcUhBzTUnVGlGhZaegdCLiM/s320/OOO_Delete_Ranges_2.tiff" /></a>
</TD>
</TR>
<TR>
<TH COLSPAN=2>
If you accidentally delete the range 'MyData' you get an error message
</TH>
<TH>
</TR><TR><TD></TD><TD>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBzJsZPTsgdjzIcQBOhb_GLltCFUMr7zojMRSw5GmCxBU2-560FWzXrL2_gqQVuyfx1UZ6PEPhdBsnMpzeSvdzfPP7Ge-q9RjKRxOy6VcXEoVwadDemROnsNakRyctmP5pcmg0vcGewFk/s1600/OOO_Delete_Range_Error.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBzJsZPTsgdjzIcQBOhb_GLltCFUMr7zojMRSw5GmCxBU2-560FWzXrL2_gqQVuyfx1UZ6PEPhdBsnMpzeSvdzfPP7Ge-q9RjKRxOy6VcXEoVwadDemROnsNakRyctmP5pcmg0vcGewFk/s300/OOO_Delete_Range_Error.tiff" /></a>
</TD>
</TR>
</TABLE>
<P>
Since my macro selects the data range it stays this way after the sort has finished.
Of course this could be changed but I view it as an indicator that something happened, the viewer is visually attracted to the marked data range.
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-79867055897440209302013-07-31T16:42:00.001+02:002013-07-31T16:57:21.415+02:00How to create a Pareto chart in OpenOffice.org or LibreOfficeUsing the data from my article <a href="http://ajhaupt.blogspot.com.es/2013/01/pareto-charts-with-google-charts.html">about creating Pareto charts with Google charts</a> I'd like to show here how to create Pareto charts in OpenOffice.org or LibreOffice.
<P>
Starting with a data set the same issues need to be resolved:
<LI>Sorting the data
<LI>Calculating the accumulated percentages
<LI>Creating the chart with data in columns, percentages as a line and two y-axes.
<P>
First lets enter the data in a spreadsheet like this:
<BR><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizZsg-f2ZF8rYNdopyKR2xjPBM4ZWx9jqw2XIJ6sDdAqJglEMkv3fc_iqVC0FGQV7CyR3xCSij_wsLE5EkSU8thYxhhrqpyGlXjA127vZkzYDS1EEy-FYG67JnEN9yFSISFKw1vI72qPM/s1600/OOo_Pareto_1.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizZsg-f2ZF8rYNdopyKR2xjPBM4ZWx9jqw2XIJ6sDdAqJglEMkv3fc_iqVC0FGQV7CyR3xCSij_wsLE5EkSU8thYxhhrqpyGlXjA127vZkzYDS1EEy-FYG67JnEN9yFSISFKw1vI72qPM/s320/OOo_Pareto_1.tiff" /></a>
<BR>
I added already the third column labeled <b>Pctg</b> which will be calculated later.
<H2>Step 1: sort the data</H2>
Mark all rows from 2 to the last and call
<b>Data -> Sort...</b> and in the <b>Sort Criteria</b> tab choose
<b>Sort key 1</b> and change the entry to
<b>Column B</b> and also tick <b>Descending</b> and finally <b>OK</b>.
<P>
<H2>Step 2: Create the Percentage values</H2>
Click on C2 and enter this formula
<CODE> = 100*SUM(B$2:B2)/SUM(B$2:B$8) </CODE>. Note that there is only one variable <b>B2</b>, the other rows are fixed.<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQBDeMOTe4as43kGeIkKe_duSeSXbTtpEjzDj27ciHa_kcFxcwfOHR7vnLzxxXYRJjniY4Y6KbdZubyWylbzvGVEtF0mEGJQYW3bVKWC1N7QYmhJNm_KsHF2WXOlrVa2MmIbX1D1oRXG8/s1600/OOo_Pareto_2.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQBDeMOTe4as43kGeIkKe_duSeSXbTtpEjzDj27ciHa_kcFxcwfOHR7vnLzxxXYRJjniY4Y6KbdZubyWylbzvGVEtF0mEGJQYW3bVKWC1N7QYmhJNm_KsHF2WXOlrVa2MmIbX1D1oRXG8/s320/OOo_Pareto_2.tiff" /></a>
<BR>
and copy it to <b>C3:C8</b> by dragging it down.
This should result in the accumulated percentages being calculated <BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtzE_8VhzKQYqSK4epSTuLcMUEVddWAkNmATxwJV92ZChEcpSSMcDlhw73clBhlheC4GGJEghkIHPl6uQ13QE7wiTmIV0y2uGV-TNDVowsg-dWxWls8FZx4xOtJnBqp3pYbRmc6wmpPdQ/s1600/OOo_Pareto_3.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtzE_8VhzKQYqSK4epSTuLcMUEVddWAkNmATxwJV92ZChEcpSSMcDlhw73clBhlheC4GGJEghkIHPl6uQ13QE7wiTmIV0y2uGV-TNDVowsg-dWxWls8FZx4xOtJnBqp3pYbRmc6wmpPdQ/s320/OOo_Pareto_3.tiff" /></a>
<BR>
<P>
Now we have all the data so we can continue with the actual chart.
<P>
<H2>Step 3: Create the basic chart</H2>
Go to <b>Insert -> Chart...</b> and in the upcoming Chart Wizard choose the chart type
<b>Column and Line</b>
with <b>Number of lines</b> set to 1 (if not already done so) and click Next.<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg69W1bE0gzH7tAzLv34O15w6KpYoP-dxobE1hVzbrmuNU_VeBIf4vCDoWnJfjypmmZ7qpAQLGJGOhdIAhdb-5CRt-lTbLH3ynCAlhWO_DcbAmFdI-ZzBGNscsrKhTY5sbtYzdRg8bCw2w/s1600/OOo_Pareto_Chart_1.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg69W1bE0gzH7tAzLv34O15w6KpYoP-dxobE1hVzbrmuNU_VeBIf4vCDoWnJfjypmmZ7qpAQLGJGOhdIAhdb-5CRt-lTbLH3ynCAlhWO_DcbAmFdI-ZzBGNscsrKhTY5sbtYzdRg8bCw2w/s320/OOo_Pareto_Chart_1.tiff" /></a>
<P>
Select the correct data range <b>$A$1:$c$8</b> by either using your mouse in the data range selector or by entering it manually. Leave the settings <b>Data series in columns</b>, <b>First row as label</b>, <b>First column as label</b> and click Next.
<br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8vjRZXdw_GRbrnKkBkZhIG6sdavt7c4stN1gpbY1VpLNWWsHWxv9om7DaY9c7vprNj4vgIuU62yOAXGFO_jktwb9mftJQqZELXmNIUSG1aCrsdAo_So9sCMzFWnu8Pij6xO184RG8Hgg/s1600/OOo_Pareto_Chart_2.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8vjRZXdw_GRbrnKkBkZhIG6sdavt7c4stN1gpbY1VpLNWWsHWxv9om7DaY9c7vprNj4vgIuU62yOAXGFO_jktwb9mftJQqZELXmNIUSG1aCrsdAo_So9sCMzFWnu8Pij6xO184RG8Hgg/s320/OOo_Pareto_Chart_2.tiff" /></a>
<P>
The following Data Series window should have everything filled in correctly, click Next.
<br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfXgBq0m1sgKBfba458rOWWBMLsue-GuAYMAAxE-EADHatla4D5K-OIQo7XfvRT2gw914uNzcStfthjAZuvTY4PJV0hFMBIB2op9HDlQSWqD_OQuVZChvZmHGNgD0QC-R9JcHGtz7Qves/s1600/OOo_Pareto_Chart_3.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfXgBq0m1sgKBfba458rOWWBMLsue-GuAYMAAxE-EADHatla4D5K-OIQo7XfvRT2gw914uNzcStfthjAZuvTY4PJV0hFMBIB2op9HDlQSWqD_OQuVZChvZmHGNgD0QC-R9JcHGtz7Qves/s320/OOo_Pareto_Chart_3.tiff" /></a>
<P>
In the last window enter titles and remove the legend:<br>
<LI>Title: Pareto chart
<LI>X axis: Category
<LI>Y axis: Sizeand click
<LI>Untick <b>Display legend</b>
<br>and click <b>Finish</b>.
<br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFUN8DYuH1nLE3BmsfqYx2tQ2chMuwyx5uTAlendQLtxd-Novu3_AjVgWV5UPtcoUWBwzKVP-vgmsZ8Ir5mu8szZIi9V3QZD18TH9lFsqxjNFnlUhjGVh65R_zCwKTOjUwsUbHjGpakdY/s1600/OOo_Pareto_Chart_4.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFUN8DYuH1nLE3BmsfqYx2tQ2chMuwyx5uTAlendQLtxd-Novu3_AjVgWV5UPtcoUWBwzKVP-vgmsZ8Ir5mu8szZIi9V3QZD18TH9lFsqxjNFnlUhjGVh65R_zCwKTOjUwsUbHjGpakdY/s320/OOo_Pareto_Chart_4.tiff" /></a>
<P>
And this is the result:<br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNYOUG8HVL2mopiy8gfOMNyGNgcQxaqWkujrJpdYPgJ1AVzJ4hR40uyZl9h82nTfQL1wGj84-pjRXn0Xctzf-1eV0X9R4iMCyFGyVIGevA0M28bZ4W6Bpi1nUONresfAabwPw0mixX5Cw/s1600/OOo_Pareto_Chart_raw.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNYOUG8HVL2mopiy8gfOMNyGNgcQxaqWkujrJpdYPgJ1AVzJ4hR40uyZl9h82nTfQL1wGj84-pjRXn0Xctzf-1eV0X9R4iMCyFGyVIGevA0M28bZ4W6Bpi1nUONresfAabwPw0mixX5Cw/s320/OOo_Pareto_Chart_raw.tiff" /></a>
<P>
What you can see: there is only one y-axis and the data and percentage units are not distinguishable.
<P>
<H2>Step 4: Fine tune the chart</H2>
First we are introducing an additonal y-axis.
Ensure the chart is marked.
<LI>Right click on the chart and choose
<b>Insert/Delete Axes...</b>.
Under <b>Secondary Axes</b> tick <b>Y axis</b> and <b>OK</b>.
This will create a second <b>identical y-axis on the right</b>.
<LI>With the chart still marked right click again and choose <b>Insert Titles...</b> and under <b>Secondary Axes</b> and <b>Y axis</b> enter <b>Pctg</b>.<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQKab1mQpzkbgnSH2s55BOAUK6YbDO-6BCxSUnR-yhyphenhyphennDISP7DdZOkze7i5JO6K9WX5ygy0KWrdmPTHt8Ea0g4_QxoqQuFRSDhlteHuBdAZ_5ucFn2TPVfWFSvO6ijjlvlORsUAyz92oI/s1600/OOo_Pareto_Chart_2nd_axis.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQKab1mQpzkbgnSH2s55BOAUK6YbDO-6BCxSUnR-yhyphenhyphennDISP7DdZOkze7i5JO6K9WX5ygy0KWrdmPTHt8Ea0g4_QxoqQuFRSDhlteHuBdAZ_5ucFn2TPVfWFSvO6ijjlvlORsUAyz92oI/s320/OOo_Pareto_Chart_2nd_axis.tiff" /></a>
<P>
Now we need to join the secondary y-axis with the line chart.
Select the red line. This can be a bit tricky, you are successful if the green selection points are shown like this<br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOvnFaOFQJCSowxhBA-Zd6suwIo0XT5MYlpanbKkiZyDdrjop1rZ2EkkVwqOZBEX00sdg2GYKljqMs1O-Qq0LPt_Y9uFO41kKMmEVYI6uAKMNtoVueEz0XHXpBgOuT6NIXLmELMNoaUv0/s1600/OOo_Pareto_Chart_2nd_axis_2.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOvnFaOFQJCSowxhBA-Zd6suwIo0XT5MYlpanbKkiZyDdrjop1rZ2EkkVwqOZBEX00sdg2GYKljqMs1O-Qq0LPt_Y9uFO41kKMmEVYI6uAKMNtoVueEz0XHXpBgOuT6NIXLmELMNoaUv0/s320/OOo_Pareto_Chart_2nd_axis_2.tiff" /></a>
<P>
Now right click on the red line and choose
<b>Format Data Series...</b>.
Select the <b>Options</b> tab and under
<b>Align data series to</b> tick
<b>Secondary Y axis</b>.
Click <b>OK</b>:<br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpF0lzau56IVJg8NHpH3fvm2izFjgKlic-5zHapOTXShdU_9Ohv90zG_kVRWrdTElchyphenhyphennDVyqlKzll9PFCrubxPj2t9RzGBSFj2i460RwbJuSf4JtnE6l1-ftnQ95_dkp7XDDVFTb4jwc/s1600/OOo_Pareto_Chart_2nd_axis_3.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpF0lzau56IVJg8NHpH3fvm2izFjgKlic-5zHapOTXShdU_9Ohv90zG_kVRWrdTElchyphenhyphennDVyqlKzll9PFCrubxPj2t9RzGBSFj2i460RwbJuSf4JtnE6l1-ftnQ95_dkp7XDDVFTb4jwc/s320/OOo_Pareto_Chart_2nd_axis_3.tiff" /></a>
<br>
The range of the secondary y-axis has changed to <b>0 - 120</b> but this automated setting is not yet what we want.
<P>
Mark the secondary y-axis e.g. by clicking on any of its numbers.
Choose <b>Format Axis...</b> and in the <b>Scale</b> tab go to <b>Maximum</b>, untick <b>Automatic</b> and <b>enter 100</b> as the maximum value.<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsbscN4D1ep-ux-Khoa2vOc5gaaxABsNIP_K-P93BzfhaaRXDQ6fA8qCXHB7IiM3pbvAWfQyiWRnX1BKCbuVLAkCHr423d7rIfeM1lpnGxzjGJSldp9VfqyfaUQ2_NOz3UPIvVA7C450Q/s1600/OOo_Pareto_Chart_2nd_axis_max_3.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsbscN4D1ep-ux-Khoa2vOc5gaaxABsNIP_K-P93BzfhaaRXDQ6fA8qCXHB7IiM3pbvAWfQyiWRnX1BKCbuVLAkCHr423d7rIfeM1lpnGxzjGJSldp9VfqyfaUQ2_NOz3UPIvVA7C450Q/s320/OOo_Pareto_Chart_2nd_axis_max_3.tiff" /></a>
<P>
This is the new graph:
<br>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitpfwosxjDP5W15IzzVx91Atwtt_jDYG7YEBvrIE3gI-hKjX9FheFlFg42pkvLuHFSjyQW323CvWL6WNWUPYwrWJV-bLIhIwnwaIwkqWYoj40Q_Mxitl9u4XtpJRczaQpLNicYNGc5YNI/s1600/OOo_Pareto_Chart_final.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitpfwosxjDP5W15IzzVx91Atwtt_jDYG7YEBvrIE3gI-hKjX9FheFlFg42pkvLuHFSjyQW323CvWL6WNWUPYwrWJV-bLIhIwnwaIwkqWYoj40Q_Mxitl9u4XtpJRczaQpLNicYNGc5YNI/s320/OOo_Pareto_Chart_final.tiff" /></a>
<P>
There are some more actions needed
to make the graph look closer to my Google chart example but I won't show all the details:
<LI>change the range of the primary y-axis to 160 and its major intervals to 40 and thus reduce the grid lines
<LI>change the major intervals of the secondary y-axis to 25
<LI>change the font attributes of each title to italic
<BR>
So here is the final result:<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjND0bdjrtAV1huMCwy8rIzJ3aDTdiBvpbe8xP2GKxaA4upa8cz3ClGD52YqGoMqzcbRQhnM26WFKDL1NpFWBxkakTkzhPRdN4krD7bLALNeiTrVI4KXnaw_lZNqd-UXeEeLj_pWrk0W4/s1600/OOo_Pareto_Chart_final_2.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjND0bdjrtAV1huMCwy8rIzJ3aDTdiBvpbe8xP2GKxaA4upa8cz3ClGD52YqGoMqzcbRQhnM26WFKDL1NpFWBxkakTkzhPRdN4krD7bLALNeiTrVI4KXnaw_lZNqd-UXeEeLj_pWrk0W4/s640/OOo_Pareto_Chart_final_2.tiff" /></a>
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com24tag:blogger.com,1999:blog-7194384172575329109.post-24075417813399119612013-07-30T21:23:00.001+02:002013-07-30T21:23:45.552+02:00LibreOffice writer: customize border of every second table rowLately I wanted to achieve the following in <b>LibreOffice writer</b>:
I had created a table and I wanted to have borders around the uneven rows whereas rows two, four should be without borders.<BR>
An example explains it quickly.
<br><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsDtpY0COY0NCuIJpB5F0y1pNqbdZI3I9shXpBfJVMIUvvkndKy-LgSzseCVGRpb4cQicvm2O1J1Ii2-uZTXpkIdI3UP5q2RYrklN6aRuI06ZYGODMcvhcyBbeNQWRt5CuCAsJjnY3QsA/s1600/Table_goal.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsDtpY0COY0NCuIJpB5F0y1pNqbdZI3I9shXpBfJVMIUvvkndKy-LgSzseCVGRpb4cQicvm2O1J1Ii2-uZTXpkIdI3UP5q2RYrklN6aRuI06ZYGODMcvhcyBbeNQWRt5CuCAsJjnY3QsA/s320/Table_goal.tiff" /></a>
<br>
Note that there is no border between cells and that the even rows don't show borders.
<P>
Here is how to do this.
<h2>Create table</H2>
The first step of course is to create a table in writer via <b>Insert -> Table</b> and choosing 5 rows.
Then fill in the cell contents as displayed above, the result being<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEGco3Jq6GItq9nVTfPBTEZn7Z764Iidr1ZApgGl5uA4TaFxOtCE4fQdO3ENBmlmd9PwSpq6Cra-52oLuXqqQSo6KR_JTeB4fR07W7aj63ESsUIfAWOF-q6yh8gTDpQ6lbXMTTbpw_3oc/s1600/Table_Insert.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEGco3Jq6GItq9nVTfPBTEZn7Z764Iidr1ZApgGl5uA4TaFxOtCE4fQdO3ENBmlmd9PwSpq6Cra-52oLuXqqQSo6KR_JTeB4fR07W7aj63ESsUIfAWOF-q6yh8gTDpQ6lbXMTTbpw_3oc/s320/Table_Insert.tiff" /></a>
<P>
<H2>Remove all borders</H2>
The next step is to clear all borders.
There are two ways to do this, I'll show one.
<BR>
Click any table cell and choose <b>Table -> Table Properties...</b>.
Click on the <b>Set No Borders</b> icon (circled red) and <b>OK</b>.<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBnttFfbb9QnQPmZy7QRzugsVCpgZxNp_DANv5vn1CbqF-HDiFLTrn_-_l-t6L_WS7PHC-d9UaMrCaTbQ85DKKSQ3OnEQtzyX9P_sM8JyzRYUqNPH4qzruMZZuUR05J59Hqtl3iyRH1_0/s1600/Table_Table_Format.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBnttFfbb9QnQPmZy7QRzugsVCpgZxNp_DANv5vn1CbqF-HDiFLTrn_-_l-t6L_WS7PHC-d9UaMrCaTbQ85DKKSQ3OnEQtzyX9P_sM8JyzRYUqNPH4qzruMZZuUR05J59Hqtl3iyRH1_0/s320/Table_Table_Format.jpg" /></a>
<P>
This will result in this table.<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiEh0Q3ixqeG6XM5UYjIeg0UsF7SN7BMXEF8B63a0XJK0coJora8jDnQhUMukFqubDL2VwVht6Hr-rIcmIGFq0n8fpymjL0nedCAfeOq-hJYvwTGZy_0XwOJFJYCrdCPNr7sEKXYXWce8/s1600/Table_Without_Borders.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiEh0Q3ixqeG6XM5UYjIeg0UsF7SN7BMXEF8B63a0XJK0coJora8jDnQhUMukFqubDL2VwVht6Hr-rIcmIGFq0n8fpymjL0nedCAfeOq-hJYvwTGZy_0XwOJFJYCrdCPNr7sEKXYXWce8/s320/Table_Without_Borders.tiff" /></a>
<P>
<H2>Adding row borders</H2>
For this and the following steps you should enable the table toolbar via <b>View -> Toolbars -> Table</b> which will display a new toolbar at the bottom of your LibreOffice window.
<P>
It is easy to <b>set the border for one row</b>:
<LI><b>Select the row</b> (either by marking all cells from left to right <i>or</i> by moving the cursor outside of the table to the left of the row until the cursor changes shape into a little arrow and clicking that arrow)
<LI>Click on the border icon (red circle) in the table toolbar to get a selection of border settings and <b>set the full border</b> (blue circle)<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhusuouHtTQp0QHYCt7AOabEsVHOwsUOlz_WX0IU-636ioELto_wtIJ5Qps0HxBj2fMG9xKFI3ShLfusdCzdgg_yh54tX70FLlT7_dwRBO0LbeTrlGLVRYJYuoTEYSrFWNoYpApQO-jTVI/s1600/Table_Set_Row_Border.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhusuouHtTQp0QHYCt7AOabEsVHOwsUOlz_WX0IU-636ioELto_wtIJ5Qps0HxBj2fMG9xKFI3ShLfusdCzdgg_yh54tX70FLlT7_dwRBO0LbeTrlGLVRYJYuoTEYSrFWNoYpApQO-jTVI/s320/Table_Set_Row_Border.jpg" /></a>
<BR>
which should result in this<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhD3NTAQt1dcKQkV0KhwVXuDLAm26T4yidDDGeJx8bQ2V0NfYSn5OtD8oHJ-4p49edZKMREnaTH7cMuo9Hx4eZEx39TrZ6swW3pbuEDFEmZMi1RkxbGBPctJ_TViyaXkFGJJ3MVEV8mP0/s1600/Table_Set_Row_Border_2.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhD3NTAQt1dcKQkV0KhwVXuDLAm26T4yidDDGeJx8bQ2V0NfYSn5OtD8oHJ-4p49edZKMREnaTH7cMuo9Hx4eZEx39TrZ6swW3pbuEDFEmZMi1RkxbGBPctJ_TViyaXkFGJJ3MVEV8mP0/s320/Table_Set_Row_Border_2.tiff" /></a>
<P>
The cumbersome thing is: one needs to do this for each row individually, in my case rows 3 and 5.
<BR>
As one can see this is<b> not feasible for big tables</b> (say 20 rows or more) and there is also no easy way to change this later (if you suddenly wanted a different border setting with no borders on the left you would need to redo these steps for all rows again).<BR>
I haven't found a nicer way to do this. Trying to mark non-consecutive rows 1, 3 and 5 did not work, in essence there doesn't seem to be a way at all in LibreOffice (or OpenOffice.org) to mark non-consecutive rows in writer (I'd be glad if someone showed me a more efficient way to set the borders).
<P>
Anyway: with this approach I finally got my table as outlined at the beginning.
<P>
<H2>Some more customizations: line style and colour</H2>
Further customizations are again quite easy.
First of all you always have to <b>select the whole table</b> (either by marking all cells <i>or</i> by clicking in the left cell in the first row and then shift clicking the left cell in the last row).
<P>
Then one could set the <b>line style</b> by clicking the line style icon (red circle) and then choosing from the list (the blue ellipse chooses the very last entry).<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg90y0TO3ufkGKjSBn7mCGJ-4D0fI21T5Z_-b1oJt66wSKGIAhiL7Tw18-tcxJFVwx53erdOKEjGaBMA_vRLY1IWbdIKlxJA9zt7SixP_lEQ2f2MW4Wq6hMucJVq6-ddQCa__5F35i4eJc/s1600/Table_Border_Line_1.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg90y0TO3ufkGKjSBn7mCGJ-4D0fI21T5Z_-b1oJt66wSKGIAhiL7Tw18-tcxJFVwx53erdOKEjGaBMA_vRLY1IWbdIKlxJA9zt7SixP_lEQ2f2MW4Wq6hMucJVq6-ddQCa__5F35i4eJc/s320/Table_Border_Line_1.jpg" /></a>
<BR>
After that the table is still marked and one could continue to set the <b>line colour</b> by clicking the colour icon (red circle) and choosing a colour (blue circle).<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIZApsW2YuyAhn-kFlM8tfQOktobCFHs1jUhsgYXbzbNYtf1rm2n5qzO-bT8rdFUr19czlv5_Zjhtupzf2q2HXieJriNXCqEOS8JwNEpmfKaDsOxUcNHsPmGPrpyMaS_36yzKxMWlGzz4/s1600/Table_Border_Color_1.jpg" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIZApsW2YuyAhn-kFlM8tfQOktobCFHs1jUhsgYXbzbNYtf1rm2n5qzO-bT8rdFUr19czlv5_Zjhtupzf2q2HXieJriNXCqEOS8JwNEpmfKaDsOxUcNHsPmGPrpyMaS_36yzKxMWlGzz4/s320/Table_Border_Color_1.jpg" /></a>
<P>
The result looks like this: a thicker border line in orange<BR>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDnbj8aWPuYFpauMeVOeO4P1P_kujbXbDdak89zbqH9gyzQAZr2QoPXETthbENO51VXXUMG62BxpTyZLIkI6jGuuYs10qYET5iyV-yqJ-EB2h_mkUUeB6t3DEohWd1jBfHGLd57lj1CH8/s1600/Table_Border_Color_2.tiff" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDnbj8aWPuYFpauMeVOeO4P1P_kujbXbDdak89zbqH9gyzQAZr2QoPXETthbENO51VXXUMG62BxpTyZLIkI6jGuuYs10qYET5iyV-yqJ-EB2h_mkUUeB6t3DEohWd1jBfHGLd57lj1CH8/s320/Table_Border_Color_2.tiff" /></a>
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-28526577129260725372013-07-15T11:10:00.000+02:002013-07-15T11:10:15.174+02:00CSS customization of blogger templateI have been asked how I achieved the coloured fonts in my code examples.
It's done via some <b>customized CSS code</b>.
<P>
My code examples are either inline using the <CODE> tag or longer code text is wrapped in <PRE> tags.
<P>
Examples:<P>
Some <CODE>inline code</CODE> and
<PRE>
some
longer
text
</PRE>
<P>
<h2>How to add CSS customization</h2>
In the blogger main menu choose
<PRE>
Template
Customize
Advanced
</PRE>
In the list of advanced customizations scroll down to
<PRE>
Add CSS
</PRE>
and in the text field enter
<PRE>
code {color:#8b0080; background:#ffffff; }
pre {color:#8b0080; background:#ffffff; border:solid 1px black; }
</PRE>
which shows customizations for <b>CODE</b> and <b>PRE</b>:
<LI>one setting for the font colour (a kind of purple)
<LI>one setting for the background colour (white)
<BR>
Additionally <b>PRE</b> gets assigned a small black border line.
<P>
I also have another customization for <b>H2 headers</b>:<BR>
<PRE>
h2 { text-transform: capitalize}
</PRE>
which capitalizes text so
<BR> This is a nice text<BR>
is shown as
<h2>This is a nice text</H2>
This example shows nicely that these <b>customizations are added on top</b> i.e. font type, font attributes (bold in this case) etc. remain as defined earlier (in the template definition by the template author).Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-85431838975551773132013-06-18T13:28:00.000+02:002013-06-18T13:28:51.836+02:00Thunderbird: How to put your signature at the top of the replySince I'm using Thunderbird as my email client (I have to manage several email accounts, business and private, and Thunderbird is the best tool in my view to achieve that) I want to customize it to best achieve my needs of course.<P>
An email reply with quoted emails arranges the answer like this.<P>
<PRE>
<TABLE BORDER>
<TR><TH>my handwritten reply</TH><TD>...</TD></TR>
<TR><TH>the quoted email</TH><TD>On .... xyz wrote:<BR><BR>Dear Andreas,<BR>...</TD></TR>
<TR><TH>my signature file</TH><TD>Best regards<BR>Andreas</TD></TR>
</TABLE>
</PRE>
<P>
What I actually wanted was a different order: the signature should be on top of the original email like this.
<PRE>
<TABLE BORDER>
<TR><TH>my handwritten reply</TH><TD>...</TD></TR>
<TR><TH>my signature file</TH><TD>Best regards<BR>Andreas</TD></TR>
<TR><TH>the quoted email</TH><TD>On .... xyz wrote:<BR><BR>Dear Andreas,<BR>...</TD></TR>
</TABLE>
</PRE>
<P>
In order to achieve that I had to adapt two settings.
<P>
<LI>First of all there is a general setting maintained in <b>Thunderbird's config file</b>. One can change that in various ways (depending on OS) and on my Mac I'll get to it via <b><i>Thunderbird -> Preferences -> Advanced</i></b> and the button <b>Config Editor ...</b> which opens the config file. There is a setting <b><i>mail.identity.default.sig_bottom</i></b> which is <b>true</b> by default and double clicking will turn it to <b>false</b>.<BR>
This is a setting for all accounts but unfortunately it does not yet achieve the desired result.
The signature still sits at the bottom of the quoted email.
The change basically was <b>just an enabler</b> to shift the signature file.
<LI>Now comes the fine tuning of the accounts.
You can decide for each account where to put the signature file and this is done by ticking a few boxes and making the right selections.<BR>
Pick the email account and go to <b><i>View settings for this account</i></b> and choose <b><i>Composition & Addressing</i></b>.
Aside from ticking the appropriate boxes the most important change was to choose where to put the signature '<b>below my reply (above the quote)</b>.
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjit8zGCGzT5OuHAilME9dZUezt0xBxNJW2TsX1HdcW141CHZdV0DONVjwcrRYv_rMiX0mbYp91bmjszPTD0j3q3Ji20tvpmbDxqfViD7WjRsPx28im0OM3x1mPpS-YLIR3hE_bqNINYXc/s1600/Sig_on_top.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjit8zGCGzT5OuHAilME9dZUezt0xBxNJW2TsX1HdcW141CHZdV0DONVjwcrRYv_rMiX0mbYp91bmjszPTD0j3q3Ji20tvpmbDxqfViD7WjRsPx28im0OM3x1mPpS-YLIR3hE_bqNINYXc/s400/Sig_on_top.png" /></a></div>
<P>
Of course I did not invent this answer but rather than having to search for it in Mozilla forums again and again (in case I forget) I put it down here as much for my own good as for the reader's who quickly stumbles upon it in a web search.
Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0tag:blogger.com,1999:blog-7194384172575329109.post-41322169726585940942013-04-30T12:42:00.003+02:002013-04-30T12:42:36.802+02:00Solaris: using pmap to identify shared and private memory of a process The <a href="http://www.oracle.com/us/products/servers-storage/solaris/solaris10/overview/index.html">Solaris</a> operating system contains a number of nice commands to explore the status of processes.<BR>
They all access the process details maintained in the process directory <b>/proc</b>.<P>
In this article I'll take a look into the
<a href="http://download.oracle.com/docs/cd/E19253-01/816-5165/pmap-1/index.html"><b>pmap</b></a> command (the link pointing to the current Solaris documentation at Oracle who inherited Solaris after acquiring Sun Microsystems)
which allows to investigate the process memory layout in various ways (refer to the link for examples).
A user can investigate his own processes, the root user can investigate any process.
<P>
<h2>pmap output for one process</H2>
In the first part of the discussion I will look into the pmap details of <b>one process</b>.
As one can see from the output below pmap can answer those questions:
<LI>How much <b>shared and private memory</b> is a process using?
<LI>Which <b>components</b> are using how much memory? One could identify libraries or the stack or whatever to be a memory eater.
<P>
I am using the <CODE> pmap -x <I>pid</I> </CODE> command to get a listing of all components and their address mapping.
<P>
Here is the <CODE> pmap -x </CODE> example for a simple 'sleep 50' command.
<PRE>
<b>647</b>: /bin/sleep 50
Address Kbytes RSS Anon Locked Mode Mapped File
08046000 8 8 4 - rw--- [ stack ]
08050000 4 4 - - r-x-- sleep
08061000 4 4 - - rw--- sleep
08062000 8 8 - - rw--- [ heap ]
D1C90000 56 24 - - r-x-- methods_unicode.so.3
D1CAD000 4 4 4 - rwx-- methods_unicode.so.3
D1CB0000 1772 36 - - r-x-- de_DE.UTF-8.so.3
D1E7A000 4 4 4 - rwx-- de_DE.UTF-8.so.3
<b>D1E80000 1080 664 - - r-x-- libc.so.1</b>
<b>D1F90000 24 12 12 - rwx-- [ anon ]</b>
<b>D1F9E000 32 32 28 - rwx-- libc.so.1</b>
<b>D1FA6000 8 8 8 - rwx-- libc.so.1</b>
<b>D1FC0000 4 4 4 - rwx-- [ anon ]</b>
D1FC4000 160 160 - - r-x-- ld.so.1
<b>D1FF0000 4 4 4 - rwx-- [ anon ]</b>
<b>D1FF4000 4 4 - - rwxs- [ anon ]</b>
D1FFC000 8 8 8 - rwx-- ld.so.1
D1FFE000 4 4 4 - rwx-- ld.so.1
-------- ------- ------- ------- -------
total Kb 3188 992 80 -
</PRE>
<P>
Some Notes:<BR>
<LI><b>647</b> in the first line is the <b>process id</b>.<BR>
<LI>The <b>RSS</b> column reports the physical memory i.e. shared and private combined.<BR>
<LI>The <b>Anon</b> column reports the <b>private</b> memory, thus <b>shared</b> can be calculated as <b>RSS - Anon</b>.<BR>
<LI>The <b>mode</b> column determines how to handle the various lines.
The <b>read</b> bit is set <b>always</b> since it does not make sense to put something into memory which cannot be read.
Looking at the write and execute bits there are these options.
<UL>
<LI> <b>r--</b>: data, read-only
<LI> <b>rw-</b>: data
<LI> <b>rwx</b>: data, executable
<LI> <b>r-x</b>: this is code, it cannot be overwritten
</UL>
Things get a little more complex when considering that certain components appear more than once.
If we look at <b>libc.so.1</b> we see three occurances, one of them in <b>mode r-x</b> (code) and the other two in <b>mode rwx</b> (writable data).
You'll also note four entries for <b>[ anon ]</b> when pmap cannot find a common name for the entry in this address space.<P>
What I want to do now is simplify and condense the pmap output by
<LI>reducing the number of columns: <b>Address</b>, <b>Kbytes</b> and <b>Locked</b> will be skipped
<LI>replacing <b>RSS</b> by a column <b>Shared</b>
<LI>replacing <b>Mode</b> by a column <b>Type</b> which holds only two possible values: <b>code</b> or <b>data</b>
<LI><b>merging all data lines</b> for a mapped file into one (eventually all r--, rw- and rwx lines will be merged into one. This is not the case in this simple example but could well happen in more complex cases.)
<P>
The table below shows calculations for <b>libc.so.1</b> and <b>[ anon ]</b> rows:
how to get from RSS and Anon to shared and private and how to merge multiple <b>data</b> lines into one.
There should be only one code line anyway so nothing needs to be done here (other than maybe introduce a check to find out if this is really the case).<P>
<CODE>
<TABLE BORDER CELLSPACING=0>
<TR><TD> </TD><TH> RSS</TH><TH> Anon</TH><TH >Shared</TH><TH> Private</TH><TH> Shared<BR>Merged</TH><TH>Private<BR>Merged</TH><TH>New name</TR>
<TR><TD>libc.so.1 <b>r-x</b></TD><TD>664</TD><TD>0</TD><TD>664</TD><TD>0</TD><TD>664</TD><TD>0</TD><TD>libc.so.1 code</TD></TR>
<TR><TD ROWSPAN=2>libc.so.1 <b>rwx</b></TD><TD>32</TD><TD>28</TD><TD>4</TD><TD>28</TD><TD ROWSPAN=2>4<BR>= 4 + 0</TD><TD ROWSPAN=2>36<BR>= 28 + 8</TD><TD ROWSPAN=2>libc.so.1 data</TD></TR>
<TR><TD>8</TD><TD>8</TD><TD>0</TD><TD>8</TD></TR>
<TR><TD ROWSPAN=4>[ anon ] rwx </TD><TD>12</TD><TD>12</TD><TD>0</TD><TD>12</TD><TD ROWSPAN=4>4<BR>=0 + 0 + 0 + 4</TD><TD ROWSPAN=4>20<BR>= 12 + 4 + 4 + 0 </TD><TD ROWSPAN=4>[ anon ] data</TD></TR>
<TR><TD>4</TD><TD>4</TD><TD>0</TD><TD>4</TD></TR>
<TR><TD>4</TD><TD>4</TD><TD>0</TD><TD>4</TD></TR>
<TR><TD>4</TD><TD>0</TD><TD>4</TD><TD>0</TD></TR>
</TABLE>
</CODE>
<P>
Here is how I want the 'pmap -x' output to look like:
<PRE>
678: /bin/sleep 50
Shared Private Type Mapped File
------------ ------------ ---- ----------
<b>4 20 data [ anon ]</b>
8 0 data [ heap ]
4 4 data [ stack ]
0 4 data de_DE.UTF-8.so.3
36 0 code de_DE.UTF-8.so.3
0 12 data ld.so.1
160 0 code ld.so.1
<b>4 36 data libc.so.1</b>
<b>664 0 code libc.so.1</b>
0 4 data methods_unicode.so.3
24 0 code methods_unicode.so.3
4 0 code sleep
4 0 data sleep
------------ ------------ ---- ----------
912 80 Total
</PRE>
Looking at the <b>total</b> line you'll see that adding shared and private <CODE>912 + 80 = 992</CODE> which is the RSS total in the original pmap output.
<P>
Here is a little <B>nawk script</B> to show how it can be done.
<PRE>
NR==1 { header = $0; # first line }
$1~/----/ { exit; # no more processing after this line }
NR>2 {
# Capture 4 columns of interest
rss = $3; if(rss=="-") rss = 0;
private = $4; if(private=="-") private = 0;
mode = substr($6,1,3);
file = $7 " " $8 " " $9 " " $10;
# Some calculations
shared = rss - private;
type = "data"; if(mode=="r-x") type = "code";
# Accumulate totals for each (file,type) combination
sharedTotal[file,type] +=shared;
privateTotal[file,type] +=private;
}
END {
if( header=="" ) exit;
print header;
printf "%12s %12s %4.4s %s\n", "Shared", "Private", "Type", "Mapped File";
printf "%12s %12s %4.4s %s\n", "------------", "------------", "----", "----------";
shared = 0; private = 0;
command = "sort +3";
for( ij in sharedTotal ) {
split(ij, a, SUBSEP);
printf "%12d %12d %4.4s %s\n", sharedTotal[ij], privateTotal[ij], a[2], a[1] | command ;
shared += sharedTotal[ij];
private += privateTotal[ij];
}
close(command);
printf "%12s %12s %4.4s %s\n", "------------", "------------", "----", "----------";
printf "%12d %12d %4.4s %s\n", shared, private, "", "Total";
}
</PRE>
<P>
Note the interesting use of the pipe in <CODE> printf "..." | command</CODE> in the 'for' loop which will sort the printed lines by mapped filename, a construct which does not exist in the old awk.
Also it is necessary to close the file descriptor <B>before</b> printing the footer lines, otherwise they would appear first and the sorted lines would be printed at the very end while finishing the program.
<P>
Of course bigger programs lead to bigger pmap output naturally e.g. firefox created more than 600 lines.
<P>
<H2>Comparing pmap for two (or more) processes</H2>
Now what you really want to do is apply this memory check to <B>all</B> of your processes and do a comparison of the totals.<P>
When you compare the entries for two different process <b>some of the mapped files will appear in both lists</b> ( libc.so.1 will probably be on each process map).
Looking at the shared and private memory there is a significant distinction.
The <b>private memory is really private</b> and belongs to just one process whereas the <b>shared memory is shared between processes</b>.
The consequences for counting memory are: private memory can simply be counted per process and the total is the sum of all whereas shared memory of two processes is
<LI>the memory in common
<LI>the shared memory used by just the first process
<LI>the shared memory used by just the second process
<BR>
In order to determine that one has to go through the list of mapped files and check for each of them whether they are unique to the process or shared with the second one.
<P>
This idea can be applied to more processes too of course.
<P>
This little shell script runs the awk script from above for every pid belonging to USER and stores its output in a file.
Another awk script prints the 'Total' line of these files and sums up the values for shared and private and print an overall total.
<PRE>
#!/bin/sh
PSLIST=`/bin/ps -u $USER -o pid | sed 1d`
[ -z "$PSLIST" ] && exit 1
# Run 'pmap -x' for each process and condense its output with the script above
for pid in $PSLIST ; do
pmap -x $pid | nawk -f pmapx.awk > pmapx.$pid
done
# Sort the filenames numerically
FILENAMES=`/bin/ls pmapx.* | sort -t. +1n`
nawk '
BEGIN { newFile = 1 }
newFile==1 {
cmd = $0;
newFile = 0;
next;
}
/^-----/ {
# The dashed lines serve as separators
pmap = ++pmap %2; # pmap alternates between 1 and 0
next
}
pmap==1 {
# There is some pmap output to be parsed
file = $4 " " $5 " " $6 " " $7;
type = $3;
# Find the biggest shared
if( $1 > shared[type,file] ) shared[type,file] = $1;
}
/Total/ {
# Use the 'Total' line to get the already accumulated private memory
private += $2;
printf "%12d %12d %s\n", $1, $2, cmd;
# Now expect a new file
newFile = 1;
}
END {
for( ij in shared )
sharedTotal += shared[ij];
printf "%12s %12s %s\n", "------------", "------------", "---------------";
printf "%12d %12d %s\n", sharedTotal, private, "Total"
}
' $FILENAMES
</PRE>
<P>
This will lead to this output (shortened a little).<BR>
First of all it lists <b>pmap errors</b> as they occur for processes which cannot be examined.<BR>
Then the totals of the condensed 'pmap -x' files are shown together with process id and name.<BR>
At the end there is a total line but - as explained above - the total shared is <b>not equal</b> to the sum of the shared memory entries in the list whereas the private total is equal to the sum of the private memory in the list.
<PRE>
pmap: cannot examine 627: permission denied
...
1048 24 828: /bin/ksh /usr/dt/bin/Xsession
2180 84 863: /usr/bin/iiimx -iiimd
2740 556 864: iiimd -nodaemon -desktop -udsfile /tmp/.iiim-andreash/:0.0 -vardir /ex
3588 2552 867: /usr/lib/gconfd-2 8
...
2312 44 920: /usr/dt/bin/sdt_shell -c unsetenv _ PWD; unsetenv DT;
1284 24 922: -csh -c unsetenv _ PWD; unsetenv DT; setenv DISPLAY :
1032 20 934: /bin/ksh /usr/dt/config/Xsession2.jds
15324 404 936: /usr/bin/gnome-session
1752 36 943: /usr/bin/gnome-keyring-daemon
3376 232 948: /usr/lib/bonobo-activation-server --ac-activate --ior-output-fd=23
5072 276 950: gnome-smproxy --sm-client-id default0
10120 436 952: /usr/lib/gnome-settings-daemon --oaf-activate-iid=OAFIID:GNOME_Setting
9556 2960 964: /usr/bin/metacity --sm-client-id=default1
15176 23648 1050: /usr/bin/gnome-terminal
...
1560 32 10640: /bin/bash /usr/bin/firefox
1588 28 10652: /bin/bash /usr/lib/firefox/run-mozilla.sh /usr/lib/firefox/firefox-bin
40628 98560 10656: /usr/lib/firefox/firefox-bin
1060 60 22265: sh
1248 48 28233: csh
1524 48 29139: vi
------------ ------------ ---------------
72260 148444 Total
</PRE>
<P>
The <b>root</b> user could run this script for all users in order to get an overview of all users.Andreashttp://www.blogger.com/profile/06112568316588988628noreply@blogger.com0