Page History

Turn Off History

How to limit memory usage of jobs

Known to work with HTCondor version: 7.8+

Users can specify the amount of memory and disk space that their job needs by setting the following attributes in their job ad.

HTCondor matches jobs with resources by referencing the above in the job's Requirements expression, and by provisioning dynamic slots with resources greater than or equal to the requested amounts.

HTCondor monitors the total virtual memory usage of jobs. This includes both physical RAM and disk-based virtual memory allocated by the job. Information about the memory and disk usage of the job is stored in the job add using the following attributes. These attributes are also available for use in STARTD policy expressions.

By default, MemoryUsage is an expression that converts ResidentSetSize to MB.

How to preempt (evict) jobs that use too much memory

Put the following in your configuration file on the execute machine. This assumes that things like PREEMPT have already been defined further up in the configuration, so put it after the other stuff, or merge it into the other stuff. 

# preempt jobs that are more than 10% over memory assigned to the slot.
PREEMPT = $(PREEMPT) || ((MemoryUsage*1.1 > Memory) =!= TRUE)
WANT_SUSPEND = $(WANT_SUSPEND) && ((MemoryUsage*1.1 > Memory) =!= TRUE)

Note that preempted jobs will go back to idle in the job queue and will potentially try to run again if they can match to a machine. If you instead wish to put the jobs on hold when they are evicted, either use the submit-side policy described later or, in HTCondor 7.3+, use the expression WANT_HOLD. One advantage of using WANT_HOLD -vs- the submit-side policy example below is the startd will evaluate these attributes much more frequently than updates are sent to the schedd. An example using WANT_HOLD : 

# hold jobs that are more than 10% over memory assigned to the slot.
MEMORY_EXCEEDED = ((MemoryUsage*1.1 > Memory) =!= TRUE)
PREEMPT = $(PREEMPT)) || $(MEMORY_EXCEEDED)
WANT_SUSPEND = $(WANT_SUSPEND) && $(MEMORY_EXCEEDED)
WANT_HOLD = $(MEMORY_EXCEEDED)
WANT_HOLD_REASON = ifThenElse( $(MEMORY_EXCEEDED), \
               "Your job used too much virtual memory.", \
               undefined )

How to hold/remove a job that uses too much memory

Jobs can hold or remove themselves by specifying a periodic_hold or periodic_remove expression. The schedd can also hold or remove jobs as dictated by the configuration expressions SYSTEM_PERIODIC_HOLD or SYSTEM_PERIODIC_REMOVE. These are all submit-side controls, whereas the PREEMPT example above is an execute-side control. One problem with the PREEMPT example is that it doesn't do a very good job of communicating to the job owner why the job was evicted. Putting the job on hold may help communicate better. Then the user knows to resubmit the job with larger memory requirements or investigate why the job used more memory than it should have. The following example configuration shows how to put jobs on hold from the submit-side when they use too much memory. All of the same issues concerning accurate measurement of working set size apply here just as they did in the PREEMPT example above. 


# When a job matches, insert the machine memory into the
# job ClassAd so periodic_remove can refer to it.
MachineMemoryString = "$$(Memory)"
SUBMIT_EXPRS = $(SUBMIT_EXPRS)  MachineMemoryString

SYSTEM_PERIODIC_HOLD = MATCH_EXP_MachineMemory =!= UNDEFINED && \
                       MemoryUsage > 0.9*int(MATCH_EXP_MachineMemoryString)

How to place a hard limit on memory usage by a job in unix

HTCondor doesn't currently provide a configuration setting for this, but you can write your own wrapper script that runs before the job and sets resource limits that are enforced by the operating system. Here is what you put in the configuration file of your execute machines: 

USER_JOB_WRAPPER = /path/to/condor_job_wrapper

The file condor_job_wrapper above can be called whatever you want. You should create that file with the following contents: 

#!/bin/sh

# change this to the maximum allowed data segment size (in kilobytes)
ulimit -d 1000000

# run the job
exec "$@"

Note that ulimit -m (maximum resident memory size) appears attractive, but it is not actually enforced on many operating systems.

Make sure the wrapper script is executable. Example: 

chmod a+x /path/to/condor_job_wrapper