Env var puzzle in csh

Today (means: some day in May 2008) I spent some time trouble shooting a strange issue until I finally got to the solution so I wrote this blog entry as an educational excercise to show the trouble shooting efforts I went through.

This was on a Solaris 10 machine.

The puzzle

I was working in a terminal window (xterm to be precise) where I had been doing some work throughout the day and I finally got to the point to set an environment variable and check its value (my login shell is csh).

foo:/home/andreas 151 % setenv THIS something
foo:/home/andreas 152 % echo T=$THIS
T=

Oh, I just set the variable and then it reports to be empty? How strange.
I checked the environment:

foo:/home/andreas 153 % env | grep THIS
THIS=something

So this looks ok, the var is definitly set but why does echo report it as empty? (and printf showed the same).

I tested other variable names but they behaved normal: echo showed their values.

I opened a new terminal window and tested this particular variable and others: all worked normal.

How can that be? What was so special about this session?

I trussed echo (for the non-Solaris users: truss is the Solaris command to inspect the behaviour of processes showing system calls etc.) and I saw that it is called without an argument, its env list shows THIS to be set though:

foo:/home/andreas 157 % truss -aeilf -r all -v all echo $THIS
26445/1:        execve("/usr/bin/echo", 0xFFBFE954, 0xFFBFE95C)  argc = 1
26445/1:         argv: echo
26445/1:         envp: USER=andreash LOGNAME=andreash HOME=/home/andreash
26445/1:          ...
26445/1:          THIS=something
...

Think about this for a while before you proceed to the next section for the solution.

The solution

I finally got to the point where I started to check other things which are set in csh: aliases and variables (not env).

The list of aliases did not show anything particular.

And then I ran set:

foo:/home/andreas 161 % set
CSHRC   /home/andreas/.cshrc
HOSTNAME    foo
...
THIS
...

So here you go: in some of my previous work that day in that particular terminal I must have set the variable THIS to empty and forgotten about it and despite setting the environment variable the first setting was still taken into account.
This is one of the traps of csh (others might call it feature, and of course this was new to me).
The Bourne shell family uses export to distinguish and this trap does not exist.

Summary

Here it is again in summary to remember:
in csh when you 'set' a variable a subsequent 'setenv' will not replace the variable in the current context.

set A=aaaaaa
setenv A bbbbbb
echo A=$A
A=aaaaaa

unset A
echo A=$A
A=bbbbbb

Going a little further: if you reverse this and first set a variable via setenv and then via set the second value will prevail

setenv A bbbbbb
set A aaaaaa
echo A=$A
A=aaaaaa

i.e. variables set with set always take precedence over variables set with setenv. Very important to remember. And only the latter will be exported to new shells though. I guess it would be a wise rule to keep a distinct set of names for set/setenv, otherwise things can get confusing (as in my starting puzzle) or things can even go wrong (when programming with false assumptions).

Conditional formatting in OpenOffice.org

What is conditional formatting? It means applying certain attributes to a spreadsheet cell (e.g. background, font size, ...) depending on some conditions being met or not. The conditions are based on the content of the cell or - in more complex cases - on other cells.

In OpenOffice.org there are two steps necessary:

• Create appropriate styles (to be chosen from)
• Set the conditional formatting for selected cells

This is different to Excel where one selects cells, sets the condition and chooses a format. In my view OpenOffice.org is more flexible because of its simplicity of change: if you want to change the format simply edit the style and the change is applied to wherever the style is being used, be it part of the conditional formatting or elsewhere. (I'm not an Excel expert though.)

Create the appropriate styles

• Click F11 or Format->Styles and Formatting
• Choose cell styles
• Move the mouse to the blank middle of the styles window and right click and press 'New...'
• Choose a style name e.g. RedBg, then choose the Background tab and set your colour, press OK or change any other attribute e.g. font type and size in the Font tab.

Do this for as many styles as you need, so that you have e.g. 3 new named styles for various backgrounds likeRedBg, GreenBg, AmberBg

Set the conditional formatting

• Select the cell or cell range
• Choose Format -> Conditional Formatting...
• Set the conditions
• Example based on numeric values of the cells
    condition 1: 'cell value is' less than 10 and select style RedBg
    condition 2: 'cell value is' less than 20 and select style AmberBg
    condition 3: 'cell value is' greater than or equal to 20 and select style GreenBg
  This also shows how to use conditions in ascending order: if the first condition is met then its style is applied. If not then the second condition will be checked aso.
• Example based on textual values of the cells
    condition 1 'formula is' exact(lower(c5);"no") and select style RedBg
    condition 2 'formula is' exact(lower(c5);"yes") and select style GreenBg
  The background will be set to either red or green depending on no/yes entries in whichever case.
  Important to note: the cell reference c5 should be the lower right hand corner of your cell range

More complex cases are to use conditional formatting based on the entries of other cells or multiple cells, you'd need to construct the conditions as formulas using the appropriate functions.

If you need more than 3 conditions I'm afraid only a macro will get you going.

Extracting strings: field delimited vs. regular expression solutions

Imagine that you have the following string describing a directory name

  tmp dir: /var/tmp/somedir

and you want to extract the directory name.

In shell scripts one can often find quick solutions like

... | awk -F: '{print $2}'
or
... | awk -F: '{print $NF}'

each leading to

 /var/tmp/somedir

(note the leading blank).

Now this solution is destined to fail if the directory name may contain a colon.

echo "  tmp dir: /var/tmp/some:dir"  | awk -F: '{print $2}'
 /var/tmp/some

echo "  tmp dir: /var/tmp/some:dir"  | awk -F: '{print $NF}'
dir

The awk solution assumes that the input line is a set of fields with a defined field delimiter which can never be part of the field values (and one could use cut or any other tool which handles field delimited input).

Though this looks nice and easy there's a safer solution which does not rely on this requirement but uses a regular expression to extract the necessary data:

s/[^:]*: //
   Skip everything up to the first colon and a subsequent space
s/[^:]*tmp dir: //
   This is more precise if one knows that there will always be a 
   string 'tmp dir: ' in the input line:
   skip everything up to the first 'tmp dir: '

   Or if you want to save the remainder use something like this
s/[^:]*: \(.*\)/\1/
   (this is for sed)

Both can be used in sed or Perl or any tool that can handle strings and regular expressions and they also have the nice side effect to remove the blank at the beginning of the resulting string.

echo "  tmp dir: /var/tmp/some:dir"  | sed 's/[^:]*: //'
/var/tmp/some:dir

echo "  tmp dir: /var/tmp/some:dir"  | perl -n -e 's/[^:]*: //; print'
/var/tmp/some:dir

The field delimiter solution seems more accessible to users and people sometimes seem to be afraid of regular expressions but the reg exp solution achieves the same and gives you more freedom (the freedom to eventually have the field delimiter as part of the value, just in case).

And an important thing is of course to test solutions if they work when your input does not follow the expected format, in the case above what happens if there is no colon in the input line.

echo "  tmp dir /var/tmp/somedir" | awk -F: '{print $2}'
     <-- There is an empty string here

echo "  tmp dir /var/tmp/somedir" | awk -F: '{print $NF}'
  tmp dir /var/tmp/somedir
     This is a copy of the original input line

echo "  tmp dir /var/tmp/somedir" | sed 's/[^:]*: //'
  tmp dir /var/tmp/somedir
     This is a copy of the original input line

echo "  tmp dir /var/tmp/somedir" | perl -n -e 's/[^:]*: //; print'
  tmp dir /var/tmp/somedir
     This is a copy of the original input line

You need to take this into account when you are using the result of this extraction further on in a script. If you want to use the reg exp solution but in case of the input line not adhering to the format you would want an empty string rather than the original line then do this:

echo "  tmp dir /var/tmp/somedir" | sed -n 's/[^:]*: //p'
     <-- There is an empty string here
     The -n ensures that sed prints only what you want, 
     the p prints the changed input line if there was a successful match,
     no match, no output

echo "  tmp dir /var/tmp/somedir" |perl -n -e 's/[^:]*: // && print'
     In Perl you simply can print if the previous substition was successful

Scope of sub processes in sh vs. ksh

Lately I ran into a problem when I started a process in the background in a Bourne shell script and the 'wait' command did not wait but the script ended immediately.
Looking into the issue it was again due to the scope of sub processes where changes made in a sub process do not influence the parent script. Moving the shell script to ksh solved the issue for me (that was on a Solaris 10 system).

So here are some thoughts about sub processes in various types of shells.

Code that works for both shells

The script starts a sleep process in the background and waits for it to finish.
The 'date' commands note the start and end times.

sh	ksh
#!/bin/sh date '+%M:%S' while : ; do sleep 10& break done wait date '+%M:%S'	#!/bin/ksh date '+%M:%S' while : ; do sleep 10& break done wait date '+%M:%S'
results in the same time difference of 10 seconds
01:24 01:34	01:24 01:34

Code that works only for ksh

Why? Because the while loop in sh is a subshell and thus any process started in it will not influence the parent shell which - in this case - has no idea that sub processes have been started at all.
So the Bourne shell script ends immediately whereas the Korn shell script waits as before.

sh	ksh
#!/bin/sh date '+%M:%S' echo | while : ; do sleep 10& break done wait date '+%M:%S'	#!/bin/ksh date '+%M:%S' echo | while : ; do sleep 10& break done wait date '+%M:%S'
results in an immediate response	results in the same behaviour as before: 10 seconds
01:24 01:24	01:24 01:34

So a while loop at the end of a command chain in Bourne shell is to be used with caution. Watch out for cmd | while ... ; do ... ; done

Another example is of course the value of variables (this is a probably better known scenario).

sh	ksh
#!/bin/sh x=parent echo | while : ; do x=child break done echo x=$x	#!/bin/ksh x=parent echo | while : ; do x=child break done echo x=$x
results in
x=parent	x=child

I found that bash behaves like sh but with ksh one has to pay attention and the situation is trickier, means depending on which ksh version you are using: the behaviour above relates to ksh on Solaris 10. The differences in Korn shell implementations make porting of ksh scripts always an issue. Some Korn shells (I think pdksh on Linux) behave in the same way as sh but I don't have access to a system so I cannot check right now.

So any coder who regularily writes for different types of shells or who needs to migrate shell scripts from one type of shell to another or even - in case of ksh - migrates shell scripts to other platforms needs to be aware of these sub process scope issues unless he risks ending up with unwanted behaviour - in the worst case destructive behaviour since the script might be doing something completely different to what it was intended to do.

Bugs or no bugs in awk, Perl, ...? Floating point arithmetic issues after all

A couple of years ago I ran into one of my biggest 'blushing red' cases in my professional career.

The starting point was some misbehaviour of awk and Perl which I reported as a bug. There weren't any bugs though, just normal behaviour which I hadn't known (when I should have, that's the blushing red). Here's the story.

You don't have to be a math wizard to tell that the following equation is true:

1.2/3 - 0.4 = 0

but awk/nawk and Perl get it wrong:

nawk 'END{ print 1.2/3-0.4}' </dev/null
-5.55112e-17

perl -e 'print 1.2/3-.4'
-5.55111512312578e-17

and even stranger

nawk 'END{ x=1.2/3; print x,x-0.4}' </dev/null
0.4 -5.55112e-17

i.e. x prints as .4 but when you subtract .4 the result is not zero.

I found more examples like 1.4/7-0.2 and first I thought it boiled down to the division by uneven prime numbers and how they are stored internally (strange thoughts one can have) and I also thought it must be a bug in some common C library.

The impact can be severe: if you set x=1.2/3-0.4 and then your code depends on whether x is less than, equal to or greater than zero you might - metaphorically speaking - miss the moon.

Eventually I created a bug report here but I got myself educated by some experts and further reading about floating point arithmetic and why the described behaviour is correct.

The bugs described above aren't bugs but a feature of how floating point works on todays computers, an area which I have neglected and never had any serious issue with in my work life.

The issue manifests in numerous ways, not just the division (my starting point), look at these examples where a simple multiplication goes wrong, pay attention to the 600.93 example:
awk

awk '{printf "%d %d %d\n", 500.93*100, 600.93*100, 700.93*100 }'
50093 60092 70093

Perl

perl -e 'printf("%d %d %d\n", 500.93*100, 600.93*100, 700.93*100)'
50093 60092 70093

C

main() {
  int x = 600.93*100;
  printf("%d\n",x);      
}

results in 60092

Why? In short because float variables are stored in binary format where certain numbers cannot be stored exactly. Any subsequent operation will also lose some of the information and thus wrong decimal results appear. When and how this information loss happens is hard to foresee unfortunately.

Is there a way around this? that dependes on the programming language. Perl offers modules like BigMath which seem to resolve this, I don't know of a general way.

The issue I'm facing is how to write safe arithmetic in awk and Perl (and who is going to look after existing scripts and rewrites them?).

Look at the awk example again replacing printf by print:

awk '{print 500.93*100, 600.93*100, 700.93*100; }'
50093 60093 70093

so this is correct now. My starting point though was an example where print provided a wrong result so the issue can manifest itself anywhere, there is no guaranteed way out of it it seems.

OpenOffice.org: platform specific code to save a file

In the last couple of days I looked into OpenOffice.org basic code to save data to a temporary file. The code should work on any platform (Unix, Windows, ...).

So I needed to find two things to get to the desired result:

• A way to determine the platform type
• A location where to store temporary files (platform dependent)
• Here's the final code snippet

A way to determine the platform type

All I could find was the getGuiType() function which returns

• 1 for Windows
• 3 for MacOS
• 4 for Unix
  and a repeated comment that this function exists for downward compatibility without explaining what the current approach should be.

Note that there is no distinction between OS versions like Windows XP vs. Windows Vista or Solaris vs. Linux.

A location where to store temporary files (platform dependent)

For MacOS and Unix systems /tmp is the obvious place to store temporary files (/var/tmp would probably also ok, even better a user specific dir like /var/tmp/$USER but one would need to ensure first that it really exists and is writable).
On Windows that is a little more complex but the environment variable TEMP seems to be set for all users (different though for each user) and thus a good choice.

Code snippet

' This code creates a temporary file called 'data.lst'
' with a path which works for each OS
Dim sUrl As String
Dim sFile As String
sFile = "/data.lst"
If GetGuiType = 1 Then
  sUrl = "file:" + Environ("TEMP") + sFile
Else
  sUrl = "file:///tmp" + sFile
End If

' Now create some data and store them into the temp file
Dim myData()    'Array to store some data
' Fill myData with contents assuming there are n data points
' ReDim myData(n)
' myData(0) = ...
' myData(1) = ...
' ...

' Call function SaveDataToFile from the OpenOffice.org Macros library 'Tools -> UCB'
SaveDataToFile(sUrl,myData())

A safer approach would be to use an environment variable (e.g. TEMP) for any OS, check whether this env var exists and the value is a writable directory and then use it but safer means (as almost always) more coding. /tmp was invented for the lazy :-)

Is this string a number or an integer? Solutions with expr, ksh and Perl

Taking my experiments with expr posted previously a little further I thought what about writing shell functions which determine whether a given string is a number or an integer?

I had already solutions in ksh for that but wanted to see them with expr (which would be a true Bourne shell solution) and also I did them in Perl.

A number is meant to be a sequence of digits ie. 1234.
An integer is either a number or a number prefixed with a minus sign ie. 1234 or -1234.

#!/bin/sh

# Test if arg $1 is a number
########################################

isNum1() {
    ksh -c "[[ \"$1\" = +([0-9]) ]]"
    return $?
}
isNum2() {
    [ `expr "$1" : '[0-9][0-9]*$'` = "0" ] && return 1
    return 0
}
isNum3() {
    perl -e 'exit 1 unless($ARGV[0]=~/^\d+$/)' -- "$1"
}

# Test if arg $1 is an integer
########################################
isInt1() {
    ksh -c "[[ \"$1\" = *(-)+([0-9]) ]]"
}
isInt2() {
    [ `expr "$1" : '[0-9][0-9]*$'` = "0" -a `expr "$1" : '-[0-9][0-9]*$'` = "0" ] && return 1
    return 0
}
isInt3() {
    perl -e 'exit 1 unless($ARGV[0]=~/^-?\d+$/)' -- "$1"

    # Here's an alternative, better to read in Perl maybe but two commands and a pipe:
    #   echo "$1" | perl -n -e 'exit 1 unless(/^-?\d+$/)'
}

# Test suite
for i in 204 -13 +88 1-2 4+5 46.09 -7.2 a abc 2x -2x t56 -t5 "2 4"
do
  isNum1 "$i" && echo Num1 $i
  isNum2 "$i" && echo Num2 $i
  isNum3 "$i" && echo Num3 $i
  isInt1 "$i" && echo Int1 $i
  isInt2 "$i" && echo Int2 $i
  isInt3 "$i" && echo Int3 $i
done

Executing the script results in
Num1 204
Num2 204
Num3 204
Int1 204
Int2 204
Int3 204
Int1 -13
Int2 -13
Int3 -13

i.e. only the first entry is a number and the first two entries are correctly identified as integers.

One drawback to expr is that it supports only basic regular expressions i.e. some useful special characters like + or '?' cannot be used and thus ksh and Perl provide more concise solutions to the above problem.

Using 'expr' in scripts

As many scriptors know (and probably hate) Bourne shell doesn't have inbuilt arithmetic capabilities so one has to resort to expr for calculations, the most famous maybe being the loop increase:

i=0
while [ $i -lt 10 ] ; do
  ...
  i=`expr $i + 1`
done

expr has more operators though than just the basic arithmetics and I don't see them used very frequently, I think I haven't used them at all so - stumbling upon it accidentally - I thought I'd play with it a little to get a better understanding and here's the result.

The match operator (string comparison with regular expressions)

The operator to compare a string to a regular expression is the colon (:).
expr will return the number of bytes matched (the curious might look into the xpg4 version of expr which returns the number of characters matched).

Also important: the regular expression always starts to compare at the beginning of the string so as if one would have used ^.

A few examples.

f="/a/c"

# does $f match ^a ? No.
expr $f : a
0

# does $f match ^/a ? Yes.
expr $f : /a
2

# does $f contain an 'a' ? Yes.
expr $f : '.*a'
4

# does $f contain a 'b' ? No.   (the regexp must be enclosed in simple quotes here)
expr $f : '.*b'
0

# does $f end with a 'c' ? Yes.
expr $f : '.*c$'
4

# does $f end with a 'b' ? No.
expr $f : '.*b$'
0

All of these examples can be used in a decision process to check whether the result is zero (no match) or not.

x=`expr ... : ...`
if [ $x -eq 0 ] ; then
  : # no match
else
  : # match
fi

To make the code a little safer one has to consider that the string to be matched might contain white space. The examples above will fail so one needs to use double quotes.

f="a b c"
expr $f : a
expr: syntax error
# since this translates to    expr a b c : a    which does not compute

# Double quotes around the string do help
expr "$f" : a
1

And even more useful is to use the extraction reg exp \1: instead of returning the number of bytes in a match one gets a string (if successful) or an empty string.

f="/ab/cd/efg"

# Extract the filename 
expr "$f" : '.*/\(.*\)'
efg

# Extract the dirname
expr "$f" : '\(.*\)/.*'
/ab/cd

f="abcdefg"

expr "$f" : '.*/\(.*\)'
        <---- # Note: this is an empty string here !!!

expr "$f" : '\(.*\)/.*'
        <---- # Note: this is an empty string here !!!

# Why empty strings? because we were trying to match a slash which is not present in $f
# Why empty strings and not 0? because we requested a string between (...)

# Now what if we wanted to solve the following: 
# if $f contains a slash then the filename is everything after the slash
# if $f does not contain a slash it should be considered a filename
# Rather than using  if ... else ... fi we can use expr, read on.

The 'or' and 'and' operator

The or operator is | and the and operator is &, both have to be escaped always.
They can be used to compare two expressions.

or: the first expression is evaluated. If it is NULL (i.e. empty or non-existing) or 0 the second expression will be evaluated too.

a=111
b=0
c=
e=555

# In the 4  comparisons below the second expression is always valid
expr "$a" \| "$e"
111

expr "$b" \| "$e"
555

expr "$c" \| "$e"
555

expr "$d" \| "$e"
555

# Here we compare a 0 value with an empty value
expr "$b" \| "$c"
0

# Here we compare an empty value to a non-existing one
#   The result of expr is also 0
expr "$c" \| "$d"
0 

and: both expressions are evaluated. If any of them is NULL or 0 then 0 is returned. Otherwise the first expression.

a=111
b=0
c=
e=555

# In the 4  comparisons below the second expression is always valid
expr "$a" \& "$e"
111

expr "$b" \& "$e"
0

expr "$c" \& "$e"
0

expr "$d" \& "$e"
0

Combining 'match' and 'and/or'

One can use these in combination to solve the emptry string issue above and a default value can be assigned, it is important to remember that expr evaluates left to right.

f="abcdefg"

expr "$f" : '.*/\(.*\)' \| "$f"
abcdefg

# These are two operations, one 'match' and one 'or':   expr   ... : ... \| ...
# If the match operation fails   "$f" : '.*/\(.*\)'    
# then evaluate the right hand side of 'or' and make this the result of 'expr'.
#   So: no slash found in $f then return the original string.

Shell argument handling: $* vs. $@

I never can recall the difference between $* and $@ in scripts so here's a little experiment which should help, hope that I remember this time :-)

First of all there is this little script to show all args of a script, call it args.sh

#!/bin/sh cnt=0 for arg do cnt=`expr $cnt + 1` printf "arg$cnt =\t$arg\n" done

args.sh 1 "2 3" "4 \"5\" 6" will show

arg1 = 1 arg2 = 2 3 arg3 = 4 "5" 6

i.e. the script counts and shows three arguments.

In order to test how $* and $@ are passed to sub processes there is a little script test.sh in four variations, once pure and once surrounded by quotes:

#!/bin/sh args.sh $*	#!/bin/sh args.sh "$*"	#!/bin/sh args.sh $@	#!/bin/sh args.sh "$@"
test.sh 1 "2 3" "4 \"5\" 6" will lead to
arg1 = 1 arg2 = 2 arg3 = 3 arg4 = 4 arg5 = "5" arg6 = 6	arg1 = 1 2 3 4 "5" 6	arg1 = 1 arg2 = 2 arg3 = 3 arg4 = 4 arg5 = "5" arg6 = 6	arg1 = 1 arg2 = 2 3 arg3 = 4 "5" 6

i.e. only the last invocation does what we want: it keeps and forwards the original arguments.

All other invocations either split the arguments by space or concatenate all arguments into one big single argument.

Another little trap I fell into: when trying to save the arguments in a variable neither invocation is successful

#!/bin/sh a=$* b="$*" c=$@ d="$@" args.sh $a args.sh $b args.sh $c args.sh $d	#!/bin/sh a=$* b="$*" c=$@ d="$@" args.sh "$a" args.sh "$b" args.sh "$c" args.sh "$d"
all lead to the complete split	all lead to a single argument
arg1 = 1 arg2 = 2 arg3 = 3 arg4 = 4 arg5 = "5" arg6 = 6	arg1 = 1 2 3 4 "5" 6

This is all in Bourne shell (if in doubt).