Page History

Turn Off History

How to generate a core file unobtrusively

This method is not nearly as powerful as some of the other options, but it has one big advantage: you can get a core file from a running daemon without killing it and restarting. It just interrupts the daemon for as long as it takes to write out the core file. 

cd /path/to/space/for/core/file
gdb -p <pid of daemon>
gcore
quit

Now you have a core file. You can load it up in gdb and walk around in data structures to see if some of them are larger than expected. You can also run 'strings' on it to see if anything unexpected jumps out. For example, if a particular string is repeated many more times than it should be, that could be a clue to help you find a leak or source of bloat. 

strings corefile | sort | uniq -c | sort -n

How to use valgrind to check a daemon for memory leaks

Valgrind is a powerful tool for catching memory leaks and sources of bloat. The main disadvantage of the tool is that it slows things down quite a lot (~20 times in my experience). You also need to stop the running daemon and restart it under valgrind.

Here is an example of how to use it. In this example, it is the collector that is being checked. 

condor_off -collector
# become the user that runs condor
sudo su root
env _CONDOR_USE_CLONE_TO_CREATE_PROCESSES=False valgrind --tool=memcheck --leak-check=yes --show-reachable=yes --leak-resolution=high >& /tmp/valgrind.log < /dev/null /path/to/condor_collector -f -p 9621 &

Note that valgrind (as of 2.4.0) crashes due to the way we use clone(), so the above example disables the clone() optimization. As of 7.1.2, condor should auto-detect that it is running under valgrind and automatically disable the clone optimization.

To check for leaks, run for a while and then do a graceful shutdown (kill -TERM). To check for bloat, kill with SIGINT instead. This will prevent it from doing a normal exit, freeing up memory, etc. That way, we can see memory that it has referenced (but which may be unexpectedly bloated). The valgrind output will contain information on the blocks of memory that were allocated at exit.

valgrind gives more detailed information when you run it with an unstripped binary (including line numbers in the call stack).

How to use valgrind to check all the daemons for memory leaks

valgrind can be told to follow all forked/execed children and where to dump its output. So we'll do just that and start up the master. You can either be root or a normal user for this, depending upon what you are testing. 

valgrind --tool=memcheck --leak-check=yes --show-reachable=yes --leak-resolution=high --log-file=/tmp/valgrind.log --trace-children=true --num-callers=16 condor_master

After the daemons have come up, run the condor_tests test suite.

How to use the google heap profiler

The google heap profiler is part of the google-perftools package, which may be easily downloaded and compiled. This example used version 0.98.

The heap profiler is useful for watching heap usage over time. Although the package documentation recommends linking the application with the tcmalloc library, I have had success with the LD_PRELOAD mechanism.

This example shows how I used the google heap profiler on the startd: 

# from the central manager
condor_off -startd
# back on the startd node
# become the user who runs condor
ksu
env LD_PRELOAD=/p/condor/workspaces/danb/google-perftools-0.98-rh5/lib/libtcmalloc.so HEAPPROFILE=/tmp/startd.hprof HEAP_PROFILE_ALLOCATION_INTERVAL=5000000  /unsup/condor/sbin/condor_startd -f >& /tmp/startd.hprof.log < /dev/null &

Every 5M of heap allocation will result in a new file /tmp/startd.hprof.000x.heap. These files can be analyzed using the pprof tool from the google-perftools. In addition to looking at individual snapshots, you can look at the difference between two files. One nice way to look at the output is with kcachegrind, the callgrind GUI. Example: 

pprof --callgrind /path/to/condor_startd /tmp/startd.hprof.0001.heap > /tmp/startd.0001.callgrind
kcachegrind /tmp/startd.0001.callgrind

Then you can look at the call tree and see which paths allocated lots of heap, hopefully leading you to discover the source of trouble. If you configure kcachegrind to know where the source code is, you can even pull up a panel showing where in the code the allocations happened.

One thing to beware of is that any code compiled with -fomit-frame-pointer may be untracked by the heap profiler. For example, openssl is usually compiled this way and I have found that the pprof --text report shows some things allocated in libcrypt code to show up as allocations in default_malloc_ex() but the pprof --callgrind report does not show this memory allocation at all. By turning off the compiler option when compiling openssl, I was able to get things to work as expected. This same issue may affect other memory profiling tools too.

How to use gdb to trap memory allocations

I mention this only because it is a clever idea from Greg Thain that might be helpful in some situation. In practice, I have found it to slow down the application so much that timeouts happen and normal operation ceases. Too bad!

The idea is to attach to a running daemon and log all of the memory allocation calls that are being made. After sufficient time, you can detach and let the program continue running.

Here's a gdb script that we used to watch the 64-bit collector: 

add-symbol-file /tmp/condor_collector.dbg
set pagination off
break malloc
commands 1
  where
  finish
end
define hookpost-finish
  info reg rax
  continue
end
break free
commands 2
  info reg rdi
  continue
end
continue

With the above commands in a file named gdb_malloc_script, you can then attach to the daemon with the following command: 

gdb gdb_malloc_script /path/to/collector_malloc.log -p <pid>

Stopping it is a bit annoying. You have to kill gdb and then kill -CONT the program if it is in T state.

With a suitable script for analyzing the malloc log, you could then see things like places in the code that allocated a lot of new memory during the time you were watching.

Attachments: