How To Allow Some Jobs

Not logged in

• Browse
• Help
• Home
• Login
• Reports
• Search
• Timeline
• Wiki

• Contents
• No-Diff
• Text

 {section: How to allow some jobs to claim the whole machine instead of one slot}
 
-Known to work with Condor version: 7.2
+Known to work with Condor version: 7.4
 
-The simplest way to achieve this is to simply set NUM_CPUS=1 so that each machine just advertises a single slot.  However, this prevents you from supporting a mix of single-cpu and whole-machine jobs.  The following example achieves the goal of supporting both in all but one respect: the Condor accountant does not charge the whole-machine user for claiming all of the slots: it only charges the user for claiming one slot.
+The simplest way to achieve this is to simply set NUM_CPUS=1 so that each machine just advertises a single slot.  However, this prevents you from supporting a mix of single-cpu and whole-machine jobs.  The following example supports both single-cpu and whole-machine jobs.
 
 First, you would have whole-machine jobs advertise themselves as such with something like the following in the submit file:
 
@@ -10,55 +10,101 @@
 +RequiresWholeMachine = True
 {endcode}
 
-Then put the following in your Condor configuration file.  Make sure it either comes after the other attributes that this appends to (such as START) or that you merge the definitions together.
+The job should also require that it run on a whole-machine if this is a requirement.  The following example to be put in the submit file should be merged together with whatever other requirements the job may have.
 
 {code}
+Requirements = IsWholeMachineSlot
+{code}
 
-# require that whole-machine jobs only match to Slot1
-START = ($(START)) && (TARGET.RequiresWholeMachine =!= TRUE || SlotID == 1)
-
-# have the machine advertise when it is running a whole-machine job
-STARTD_JOB_EXPRS = $(STARTD_JOB_EXPRS) RequiresWholeMachine
-
-# Export the job expr to all other slots
-STARTD_SLOT_EXPRS = RequiresWholeMachine
+Then put the following in your Condor configuration file.  Make sure it either comes after the attributes to which this policy appends (such as START) or that you merge the definitions together.
 
-# require that no single-cpu jobs may start when a whole-machine job is running
-START = ($(START)) && (SlotID == 1 || Slot1_RequiresWholeMachine =!= True)
+{code}
 
-# suspend existing single-cpu jobs when there is a whole-machine job
-SUSPEND = ($(SUSPEND)) || (SlotID != 1 && Slot1_RequiresWholeMachine =?= True)
+# we will double-allocate resources to overlapping slots
+NUM_CPUS = $(DETECTED_CORES)*2
+MEMORY = $(DETECTED_MEMORY)*2
+
+# single-core slots get 1 core each
+SLOT_TYPE_1 = cpus=1
+NUM_SLOTS_TYPE_1 = $(DETECTED_CORES)
+
+# whole-machine slot gets as many cores and RAM as the machine has
+SLOT_TYPE_2 = cpus=$(DETECTED_CORES), mem=$(DETECTED_MEMORY)
+NUM_SLOTS_TYPE_2 = 1
+
+# Macro specifying the slot id of the whole-machine slot
+# Example: on an 8-core machine, the whole-machine slot is 9.
+WHOLE_MACHINE_SLOT = ($(DETECTED_CORES)+1)
+
+# ClassAd attribute that is True/False depending on whether this slot is
+# the whole-machine slot
+IsWholeMachineSlot = SlotID == $(WHOLE_MACHINE_SLOT)
+STARTD_EXPRS = $(STARTD_EXPRS) IsWholeMachineSlot
+
+# advertise state of each slot as SlotX_State in ClassAds of all other slots
+STARTD_SLOT_EXPRS = $(STARTD_SLOT_EXPRS) State
+
+# Macro for referencing state of the whole-machine slot.
+# Relies on eval(), which was added in Condor 7.3.2.
+WHOLE_MACHINE_SLOT_STATE = \
+  eval(strcat("Slot",$(WHOLE_MACHINE_SLOT),"_State"))
+
+# Macro that is true if any single-core slots are claimed
+# WARNING: THERE MUST BE AN ENTRY FOR ALL SINGLE-CORE SLOTS
+# IN THE EXPRESSION BELOW.  If you have more slots, you must
+# extend this expression to cover them.  If you have fewer
+# slots, extra entries are harmless.
+SINGLE_CORE_SLOTS_CLAIMED = \
+  0 == \
+  - $(WHOLE_MACHINE_SLOT_STATE) =?= "Claimed" + \
+  Slot1_State =?= "Claimed" + \
+  Slot2_State =?= "Claimed" + \
+  Slot3_State =?= "Claimed" + \
+  Slot4_State =?= "Claimed" + \
+  Slot5_State =?= "Claimed" + \
+  Slot6_State =?= "Claimed" + \
+  Slot7_State =?= "Claimed" + \
+  Slot8_State =?= "Claimed" + \
+  Slot9_State =?= "Claimed" + \
+  Slot10_State =?= "Claimed" + \
+  Slot11_State =?= "Claimed" + \
+  Slot12_State =?= "Claimed" + \
+  Slot13_State =?= "Claimed" + \
+  Slot14_State =?= "Claimed" + \
+  Slot15_State =?= "Claimed" + \
+  Slot16_State =?= "Claimed"
+
+# Single-core jobs must run on single-core slots
+START_SINGLE_CORE_JOB = \
+  TARGET.RequiresWholeMachine =!= True && MY.IsWholeMachineSlot == False && \
+  $(WHOLE_MACHINE_SLOT_STATE) =!= "Claimed"
+
+# Whole-machine jobs must run on the whole-machine slot
+START_WHOLE_MACHINE_JOB = \
+  TARGET.RequiresWholeMachine =?= True && MY.IsWholeMachineSlot
+
+START = ($(START)) && ( \
+  ($(START_SINGLE_CORE_JOB)) || \
+  ($(START_WHOLE_MACHINE_JOB)) )
+
+# Suspend the whole-machine job until single-core jobs finish.
+SUSPEND = ($(SUSPEND)) || ( \
+  MY.IsWholeMachineSlot && ($(SINGLE_CORE_SLOTS_CLAIMED)) )
 
 CONTINUE = ( $(SUSPEND) =!= True )
 
-{endcode}
-
-Instead of suspending the single-cpu jobs while the whole-machine job runs, you could suspend the whole-machine job while the single-cpu jobs finish.  Example:
+WANT_SUSPEND = ($(WANT_SUSPEND)) || ($(SUSPEND))
 
-{code}
-# advertise the activity of each slot into the ads of the other slots,
-# so the SUSPEND expression can see it
-STARTD_SLOT_EXPRS = $(STARTD_SLOT_EXPRS) Activity
-
-# Suspend the whole-machine job until the other slots are empty
-SUSPEND = ($(SUSPEND)) || (SlotID == 1 && Slot1_RequiresWholeMachine =?= True && \
-           (Slot2_Activity =?= "Busy" || Slot3_Activity =?= "Busy" || ... ) )
 {endcode}
 
-
-You might want to steer whole-machine jobs towards machines that are completely vacant, especially on the slots only for single-cpu jobs.
-
-Here's a simple example that just avoids machines with a high load:
+Instead of suspending the whole-machine job until single-cpu jobs finish, we could instead suspend single-core jobs while whole-machine jobs run.  Replacing the SUSPEND expression above with the following would achieve this policy.  Beware that this could cause single-core jobs to be suspended indefinitely if there is a steady supply of whole-machine jobs.
 
 {code}
-NEGOTIATOR_PRE_JOB_RANK = -TARGET.LoadAvg*(MY.RequiresWholeMachine =?= True)
+# Suspend single-core jobs while the whole-machine job runs
+SUSPEND = ($(SUSPEND)) || ( \
+  MY.IsWholeMachineSlot =!= True && $(WHOLE_MACHINE_SLOT_STATE) =?= "Claimed" )
 {endcode}
 
-A more complicated expression would look at the attributes of the other slots when forming the rank:
+{subsection: Accounting}
 
-{code}
-STARTD_SLOT_EXPRS = $(STARTD_SLOT_EXPRS) Activity
-
-NEGOTIATOR_PRE_JOB_RANK = (MY.RequiresWholeMachine =?= True) * \
-                          (Slot2_Activity =!= "Busy" + Slot3_Activity =!= "Busy" + ... )
-{endcode}
+The above policies rely on job suspension.  Should the jobs be "charged" for the time they spend in a suspended state?  This affects the user's fair-share priority and the accumulated number of hours reported by condor_userprio.  As of Condor 7.4, the default behavior is to charge the jobs for time they spend in a suspended state.  There is a configuration variable, NEGOTIATOR_DISCOUNT_SUSPENDED_RESOURCES that can be used to get the opposite behavior.

This work is supported by NSF under Cooperative Agreement OAC-2030508 as part of the PATh Project. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF. Site built using CVSTrac.