Tuesday, April 30, 2013

Solaris: using pmap to identify shared and private memory of a process

The Solaris operating system contains a number of nice commands to explore the status of processes.
They all access the process details maintained in the process directory /proc.

In this article I'll take a look into the pmap command (the link pointing to the current Solaris documentation at Oracle who inherited Solaris after acquiring Sun Microsystems) which allows to investigate the process memory layout in various ways (refer to the link for examples). A user can investigate his own processes, the root user can investigate any process.

pmap output for one process

In the first part of the discussion I will look into the pmap details of one process. As one can see from the output below pmap can answer those questions:
  • How much shared and private memory is a process using?
  • Which components are using how much memory? One could identify libraries or the stack or whatever to be a memory eater.

    I am using the  pmap -x pid  command to get a listing of all components and their address mapping.

    Here is the  pmap -x  example for a simple 'sleep 50' command.

    647:    /bin/sleep 50
     Address  Kbytes     RSS    Anon  Locked Mode   Mapped File
    08046000       8       8       4       - rw---    [ stack ]
    08050000       4       4       -       - r-x--  sleep
    08061000       4       4       -       - rw---  sleep
    08062000       8       8       -       - rw---    [ heap ]
    D1C90000      56      24       -       - r-x--  methods_unicode.so.3
    D1CAD000       4       4       4       - rwx--  methods_unicode.so.3
    D1CB0000    1772      36       -       - r-x--  de_DE.UTF-8.so.3
    D1E7A000       4       4       4       - rwx--  de_DE.UTF-8.so.3
    D1E80000    1080     664       -       - r-x--  libc.so.1
    D1F90000      24      12      12       - rwx--    [ anon ]
    D1F9E000      32      32      28       - rwx--  libc.so.1
    D1FA6000       8       8       8       - rwx--  libc.so.1
    D1FC0000       4       4       4       - rwx--    [ anon ]
    D1FC4000     160     160       -       - r-x--  ld.so.1
    D1FF0000       4       4       4       - rwx--    [ anon ]
    D1FF4000       4       4       -       - rwxs-    [ anon ]
    D1FFC000       8       8       8       - rwx--  ld.so.1
    D1FFE000       4       4       4       - rwx--  ld.so.1
    -------- ------- ------- ------- -------
    total Kb    3188     992      80       -
    
    

    Some Notes:

  • 647 in the first line is the process id.
  • The RSS column reports the physical memory i.e. shared and private combined.
  • The Anon column reports the private memory, thus shared can be calculated as RSS - Anon.
  • The mode column determines how to handle the various lines. The read bit is set always since it does not make sense to put something into memory which cannot be read. Looking at the write and execute bits there are these options.
    • r--: data, read-only
    • rw-: data
    • rwx: data, executable
    • r-x: this is code, it cannot be overwritten
    Things get a little more complex when considering that certain components appear more than once. If we look at libc.so.1 we see three occurances, one of them in mode r-x (code) and the other two in mode rwx (writable data). You'll also note four entries for [ anon ] when pmap cannot find a common name for the entry in this address space.

    What I want to do now is simplify and condense the pmap output by

  • reducing the number of columns: Address, Kbytes and Locked will be skipped
  • replacing RSS by a column Shared
  • replacing Mode by a column Type which holds only two possible values: code or data
  • merging all data lines for a mapped file into one (eventually all r--, rw- and rwx lines will be merged into one. This is not the case in this simple example but could well happen in more complex cases.)

    The table below shows calculations for libc.so.1 and [ anon ] rows: how to get from RSS and Anon to shared and private and how to merge multiple data lines into one. There should be only one code line anyway so nothing needs to be done here (other than maybe introduce a check to find out if this is really the case).

    RSS AnonShared Private Shared
    Merged
    Private
    Merged
    New name
    libc.so.1 r-x664066406640libc.so.1 code
    libc.so.1 rwx32284284
    = 4 + 0
    36
    = 28 + 8
    libc.so.1 data
    8808
    [ anon ] rwx 12120124
    =0 + 0 + 0 + 4
    20
    = 12 + 4 + 4 + 0
    [ anon ] data
    4404
    4404
    4040

    Here is how I want the 'pmap -x' output to look like:

    678:  /bin/sleep 50
          Shared      Private Type Mapped File
    ------------ ------------ ---- ----------
               4           20 data [ anon ]
               8            0 data [ heap ]
               4            4 data [ stack ]
               0            4 data de_DE.UTF-8.so.3
              36            0 code de_DE.UTF-8.so.3
               0           12 data ld.so.1
             160            0 code ld.so.1
               4           36 data libc.so.1
             664            0 code libc.so.1
               0            4 data methods_unicode.so.3
              24            0 code methods_unicode.so.3
               4            0 code sleep
               4            0 data sleep
    ------------ ------------ ---- ----------
             912           80      Total
    
    Looking at the total line you'll see that adding shared and private 912 + 80 = 992 which is the RSS total in the original pmap output.

    Here is a little nawk script to show how it can be done.

    NR==1     { header = $0; # first line }
    $1~/----/ { exit;        # no more processing after this line }
    NR>2      {
      # Capture 4 columns of interest
      rss = $3;     if(rss=="-")     rss = 0;
      private = $4; if(private=="-") private = 0;
      mode = substr($6,1,3);
      file = $7 " " $8 " " $9 " " $10;
    
      # Some calculations
      shared = rss - private;
      type   = "data"; if(mode=="r-x") type = "code";
    
      # Accumulate totals for each (file,type) combination
      sharedTotal[file,type] +=shared;
      privateTotal[file,type] +=private;
    }
    
    END {
      if( header=="" ) exit;
      print header;
      printf "%12s %12s %4.4s %s\n", "Shared", "Private", "Type", "Mapped File";
      printf "%12s %12s %4.4s %s\n", "------------", "------------", "----", "----------";
    
      shared = 0; private = 0;
      command = "sort +3";
      for( ij in sharedTotal ) {
        split(ij, a, SUBSEP);
        printf "%12d %12d %4.4s %s\n", sharedTotal[ij], privateTotal[ij], a[2], a[1] | command ;
        shared += sharedTotal[ij];
        private += privateTotal[ij];
      }
      close(command);
    
      printf "%12s %12s %4.4s %s\n", "------------", "------------", "----", "----------";
      printf "%12d %12d %4.4s %s\n", shared, private, "", "Total";
    
    }
    

    Note the interesting use of the pipe in printf "..." | command in the 'for' loop which will sort the printed lines by mapped filename, a construct which does not exist in the old awk. Also it is necessary to close the file descriptor before printing the footer lines, otherwise they would appear first and the sorted lines would be printed at the very end while finishing the program.

    Of course bigger programs lead to bigger pmap output naturally e.g. firefox created more than 600 lines.

    Comparing pmap for two (or more) processes

    Now what you really want to do is apply this memory check to all of your processes and do a comparison of the totals.

    When you compare the entries for two different process some of the mapped files will appear in both lists ( libc.so.1 will probably be on each process map). Looking at the shared and private memory there is a significant distinction. The private memory is really private and belongs to just one process whereas the shared memory is shared between processes. The consequences for counting memory are: private memory can simply be counted per process and the total is the sum of all whereas shared memory of two processes is

  • the memory in common
  • the shared memory used by just the first process
  • the shared memory used by just the second process
    In order to determine that one has to go through the list of mapped files and check for each of them whether they are unique to the process or shared with the second one.

    This idea can be applied to more processes too of course.

    This little shell script runs the awk script from above for every pid belonging to USER and stores its output in a file. Another awk script prints the 'Total' line of these files and sums up the values for shared and private and print an overall total.

    #!/bin/sh
    
    PSLIST=`/bin/ps -u $USER -o pid | sed 1d`
    [ -z "$PSLIST" ] && exit 1
    
    # Run 'pmap -x' for each process and condense its output with the script above
    for pid in $PSLIST  ; do
      pmap -x $pid | nawk -f pmapx.awk > pmapx.$pid
    done
    
    # Sort the filenames numerically
    FILENAMES=`/bin/ls pmapx.* | sort -t. +1n`
    
    nawk '
    BEGIN { newFile = 1 }
    newFile==1 { 
      cmd = $0; 
      newFile = 0;
      next;
    }
    /^-----/ {
      # The dashed lines serve as separators
      pmap = ++pmap %2;  # pmap alternates between 1 and 0
      next
    }
    pmap==1 {
      # There is some pmap output to be parsed
      file = $4 " " $5 " " $6 " " $7;
      type = $3;
      # Find the biggest shared
      if( $1 > shared[type,file] ) shared[type,file] = $1;
    }
    /Total/ {
      # Use the 'Total' line to get the already accumulated private memory
      private += $2;
      printf "%12d %12d   %s\n", $1, $2, cmd;
      # Now expect a new file
      newFile = 1;
    }
    END {
      for( ij in shared )
        sharedTotal += shared[ij];
      printf "%12s %12s   %s\n", "------------", "------------", "---------------";
      printf "%12d %12d   %s\n", sharedTotal, private, "Total"
    }
    ' $FILENAMES
    

    This will lead to this output (shortened a little).
    First of all it lists pmap errors as they occur for processes which cannot be examined.
    Then the totals of the condensed 'pmap -x' files are shown together with process id and name.
    At the end there is a total line but - as explained above - the total shared is not equal to the sum of the shared memory entries in the list whereas the private total is equal to the sum of the private memory in the list.

    pmap: cannot examine 627: permission denied
    ...
            1048           24   828:        /bin/ksh /usr/dt/bin/Xsession
            2180           84   863:        /usr/bin/iiimx -iiimd
            2740          556   864:        iiimd -nodaemon -desktop -udsfile /tmp/.iiim-andreash/:0.0 -vardir /ex
            3588         2552   867:        /usr/lib/gconfd-2 8
    ...
            2312           44   920:        /usr/dt/bin/sdt_shell -c unsetenv _ PWD;            unsetenv DT;
            1284           24   922:        -csh -c unsetenv _ PWD;             unsetenv DT;      setenv DISPLAY :
            1032           20   934:        /bin/ksh /usr/dt/config/Xsession2.jds
           15324          404   936:        /usr/bin/gnome-session
            1752           36   943:        /usr/bin/gnome-keyring-daemon
            3376          232   948:        /usr/lib/bonobo-activation-server --ac-activate --ior-output-fd=23
            5072          276   950:        gnome-smproxy --sm-client-id default0
           10120          436   952:        /usr/lib/gnome-settings-daemon --oaf-activate-iid=OAFIID:GNOME_Setting
            9556         2960   964:        /usr/bin/metacity --sm-client-id=default1
           15176        23648   1050:       /usr/bin/gnome-terminal
     ...
            1560           32   10640:      /bin/bash /usr/bin/firefox
            1588           28   10652:      /bin/bash /usr/lib/firefox/run-mozilla.sh /usr/lib/firefox/firefox-bin
           40628        98560   10656:      /usr/lib/firefox/firefox-bin
            1060           60   22265:      sh
            1248           48   28233:      csh
            1524           48   29139:      vi
    ------------ ------------   ---------------
           72260       148444   Total
    

    The root user could run this script for all users in order to get an overview of all users.

  • 3 comments:

    1. Harvard Business Review named data scientist the "sexiest job of the 21st century".This Data Science course will cover the whole data life cycle ranging from Data Acquisition and Data Storage using R-Hadoop concepts, Applying modelling through R programming using Machine learning algorithms and illustrate impeccable Data Visualization by leveraging on 'R' capabilities.With companies across industries striving to bring their research and analysis (R&A) departments up to speed, the demand for qualified data scientists is rising.

      data science training in bangalore

      ReplyDelete
    2. myTectra offers Big Data and Hadoop training in Bangalore using Class Room.
      myTectra offers Live Online Big Data and Hadoop training Globally.
      Big Data and Hadoop training Unlike traditional systems, Big Data and Hadoop enables multiple types of analytic workloads to run on the same data, at the same time, at massive scale on industry-standard hardware.myTectra Big Data and Hadoop training is designed to help you become a expert Hadoop developer. myTectra offers Big Data Hadoop Training in Bangalore using Class Room. myTectra offers Live Online Big Data and Hadoop training Globally.

      hadoop training in bangalore

      ReplyDelete
    3. Python has adopted as a language of choice for almost all the domain in IT including the most trending technologies such as Artificial Intelligence, Machine Learning, Data Science, Internet of Things (IoT), Cloud Computing technologies such as AWS, OpenStack, VMware, Google Cloud, etc.., Big Data Analytics, DevOps and Python is prepared language in traditional IT domain such as Web Application Development, Infrastructure Automation ,Software Testing, Mobile Testing.

      python online training

      ReplyDelete