Every 5M of heap allocation will result in a new file /tmp/startd.hprof.000x.heap. These files can be analyzed using the pprof tool from the google-perftools. In addition to looking at individual snapshots, you can look at the difference between two files. One nice way to look at the output is with kcachegrind, the callgrind GUI. Example:
{verbatim}
-pprof --callgrind /path/to/condor_collector /tmp/condor.prof.0001.heap > /tmp/collector.0001.callgrind
-kcachegrind /tmp/collector.0001.callgrind
+pprof --callgrind /path/to/condor_startd /tmp/startd.hprof.0001.heap > /tmp/startd.0001.callgrind
+kcachegrind /tmp/startd.0001.callgrind
{endverbatim}
Then you can look at the call tree and see which paths allocated lots of heap, hopefully leading you to discover the source of trouble. If you configure kcachegrind to know where the source code is, you can even pull up a panel showing where in the code the allocations happened.
