{section: How to limit memory usage of jobs}
-Known to work with HTCondor version: 7.0
+Known to work with HTCondor version: 7.8+
-HTCondor monitors the total virtual memory usage of jobs. This includes both physical RAM and disk-based virtual memory allocated by the job. Administrators often want to limit the amount of physical RAM used by the job so that it doesn't cause performance problems for other jobs or tasks on the computer. This is difficult to do in a general way, because HTCondor currently lacks a way to measure how much physical RAM the application actually needs verses how much of its memory could be swapped out to the disk without impacting performance. (How much the application actually needs is known as the working set size.)
+Users can specify the amount of memory and disk space that their job needs by setting the following attributes in their job ad.
-Example: HTCondor may see that the virtual memory size of a job is 1.5GB when there is only 1GB per slot on the 4 core system. This could be a problem if jobs on the other slots need their full 1GB of expected memory. However, it may be that there simply isn't demand for memory at the moment, so the operating system is letting this job keep more of its memory in physical RAM than it actually needs. If something else comes along and demands more memory, the memory usage of this job might painlessly shift so that only 1.0GB is in physical RAM and the other 0.5GB is on disk, leaving the expected amount of RAM for other jobs without causing poor performance due to thrashing (actively needed data jumping back and forth between disk and RAM).
+*: =RequestDisk= The amount of disk needed by the job in KB
+*: =RequestMemory= The amount of memory needed by the job in MB
-This is an area we hope to improve in HTCondor. In the mean time, here are some recipes that have proven useful, even though they are not perfect.
+HTCondor matches jobs with resources by referencing the above in the job's =Requirements= expression, and by provisioning dynamic slots with resources greater than or equal to the requested amounts.
+HTCondor monitors the total virtual memory usage of jobs. This includes both physical RAM and disk-based virtual memory allocated by the job. Information about the memory and disk usage of the job is stored in the job add using the following attributes
+
+*: =DiskUsage= The amount of disk used in KB
+*: =MemoryUsage= The amount of memory used in MB
+*: =ImageSize= Maximum observed memory image size in KB
+*: =ResidentSetSize= The amount of virtual memory used in KB
+*: =ProportionalSetSize= when avaliable, this is a better measure of virtual memory used for jobs that have multiple processes that share memory.
+
+By default, =MemoryUsage= is an expression that converts =ResidentSetSize= to MB.
{subsection: How to preempt (evict) jobs that use too much memory}
Put the following in your configuration file on the execute machine. This assumes that things like PREEMPT have already been defined further up in the configuration, so put it after the other stuff, or merge it into the other stuff.
{code}
-# Let a job use up to 90% of the memory allocated to its batch slot
-MEMORY_AVAILABLE_MB = (Memory*0.9)
-
-
-# The working set size is the amount of memory the job actually needs
-# in RAM (as opposed to disk-based memory) in order to run without
-# thrashing (copying data back and forth between RAM and disk frequently).
-# If the job has an attribute "MemoryRequirementsMB", then we use that
-# for the working set size. This is a custom attribute that would have
-# to be manually set by the user, and which we trust in place of our
-# default assumption. The default in this example is to arbitrarily
-# assume the working set size is 70% of the virtual memory size.
-# That will certainly be wrong if the job calls mmap() on a large file,
-# but doesn't need the full file in RAM, so in such cases, the user will
-# have to set MemoryRequirementsMB.
-
-WORKING_SET_SIZE_MB = ifThenElse( isUndefined(MemoryRequirementsMB), \
- ImageSize/1024*0.7, \
- MemoryRequirementsMB )
-
-# Here we check if the working set size of the job is greater than
-# the RAM allocated to this batch slot. We also check if the virtual
-# memory size of the job is greater than the total virtual memory allocated
-# to this batch slot. If either is true, then memory is exceeded.
-
-MEMORY_EXCEEDED = $(WORKING_SET_SIZE_MB) > $(MEMORY_AVAILABLE_MB)
-
-PREEMPT = ($(PREEMPT)) || ($(MEMORY_EXCEEDED))
-
-WANT_SUSPEND = ($(WANT_SUSPEND)) && ($(MEMORY_EXCEEDED)) =!= TRUE
-
+# preempt jobs that are more than 10% over memory assigned to the slot.
+PREEMPT = $(PREEMPT) || ((MemoryUsage*1.1 > Memory) =!= TRUE)
+WANT_SUSPEND = $(WANT_SUSPEND) && ((MemoryUsage*1.1 > Memory) =!= TRUE)
{endcode}
Note that preempted jobs will go back to idle in the job queue and will potentially try to run again if they can match to a machine. If you instead wish to put the jobs on hold when they are evicted, either use the submit-side policy described later or, in HTCondor 7.3+, use the expression WANT_HOLD. One advantage of using WANT_HOLD -vs- the submit-side policy example below is the startd will evaluate these attributes much more frequently than updates are sent to the schedd. An example using WANT_HOLD :
{code}
-VIRTUAL_MEMORY_AVAILABLE_MB = (VirtualMemory*0.9)
-MEMORY_EXCEEDED = ImageSize/1024 > $(VIRTUAL_MEMORY_AVAILABLE_MB)
-PREEMPT = ($(PREEMPT)) || ($(MEMORY_EXCEEDED))
-WANT_SUSPEND = ($(WANT_SUSPEND)) && ($(MEMORY_EXCEEDED)) =!= TRUE
-WANT_HOLD = ($(MEMORY_EXCEEDED))
-WANT_HOLD_REASON = \
- ifThenElse( $(MEMORY_EXCEEDED), \
+# hold jobs that are more than 10% over memory assigned to the slot.
+MEMORY_EXCEEDED = ((MemoryUsage*1.1 > Memory) =!= TRUE)
+PREEMPT = $(PREEMPT)) || $(MEMORY_EXCEEDED)
+WANT_SUSPEND = $(WANT_SUSPEND) && $(MEMORY_EXCEEDED)
+WANT_HOLD = $(MEMORY_EXCEEDED)
+WANT_HOLD_REASON = ifThenElse( $(MEMORY_EXCEEDED), \
"Your job used too much virtual memory.", \
undefined )
{endcode}
+
+
{subsection: How to hold/remove a job that uses too much memory}
Jobs can hold or remove themselves by specifying a periodic_hold or periodic_remove expression. The schedd can also hold or remove jobs as dictated by the configuration expressions SYSTEM_PERIODIC_HOLD or SYSTEM_PERIODIC_REMOVE. These are all submit-side controls, whereas the PREEMPT example above is an execute-side control. One problem with the PREEMPT example is that it doesn't do a very good job of communicating to the job owner why the job was evicted. Putting the job on hold may help communicate better. Then the user knows to resubmit the job with larger memory requirements or investigate why the job used more memory than it should have. The following example configuration shows how to put jobs on hold from the submit-side when they use too much memory. All of the same issues concerning accurate measurement of working set size apply here just as they did in the PREEMPT example above.
{code}
-# The working set size is the amount of memory the job actually needs
-# in RAM (as opposed to disk-based memory) in order to run without
-# thrashing (copying data back and forth between RAM and disk frequently).
-# If the job has an attribute "MemoryRequirementsMB", then we use that
-# for the working set size. This is a custom attribute that would have
-# to be manually set by the user, and which we trust in place of our
-# default assumption. The default in this example is to arbitrarily
-# assume the working set size is 70% of the virtual memory size.
-# That will certainly be wrong if the job calls mmap() on a large file,
-# but doesn't need the full file in RAM, so in such cases, the user will
-# have to set MemoryRequirementsMB.
-
-WORKING_SET_SIZE_MB = ifThenElse( isUndefined(MemoryRequirementsMB), \
- ImageSize/1024*0.7, \
- MemoryRequirementsMB )
# When a job matches, insert the machine memory into the
# job ClassAd so periodic_remove can refer to it.
@@ -88,7 +56,7 @@
SUBMIT_EXPRS = $(SUBMIT_EXPRS) MachineMemoryString
SYSTEM_PERIODIC_HOLD = MATCH_EXP_MachineMemory =!= UNDEFINED && \
- $(WORKING_SET_SIZE_MB) > 0.9*int(MATCH_EXP_MachineMemoryString)
+ MemoryUsage > 0.9*int(MATCH_EXP_MachineMemoryString)
{endcode}
{subsection: How to place a hard limit on memory usage by a job in unix}
