ParallelSchedulingGroup = "$(FULL_HOSTNAME)" STARTD_ATTRS = $(STARTD_ATTRS), ParallelSchedulingGroup
+WantParallelSchedulingGroups = True
If you don't wish to require packing all nodes of a parallel job onto the same physical machine, you could tell HTCondor to attempt to use the least number of physical machines to run a parallel job as follows:
## The NEGOTIATOR_PRE_JOB_RANK expression overrides all other ranks ## that are used to pick a match from the set of possibilities. ## The following expression matches jobs to unclaimed slots ## whenever possible, regardless of the job-supplied rank. ## For all slots, prefer to give out the highest MachineId ## available to parallel universe jobs in order to try to pack parallel ## jobs onto the the minimum number of physical machines. NEGOTIATOR_PRE_JOB_RANK =(1000000*RemoteOwner =?= UNDEFINED) \ + ( (TARGET.JobUniverse=?=11)*MachineId ) ## Now, if the job is a parallel universe job, tell condor_submit to ## append a job rank expression in the job classads that prefers to use ## the highest MachineId available. Note that for parallel universe ## jobs it is not enough to set NEGOTIATOR ranking alone to use ## the highest MachineId available, we must also put it into the job ## classad itself because the dedicated scheduler does its own ## re-sorting of jobs to machine mappings after it gets its matches ## from the negotiator. APPEND_RANK = (JobUniverse=?=11)*MachineId
The above example assumes you have a global unique ordering of your physical machines as an integer in machine ad attribute MachineId. If you cannot easily add such a global ordering, perhaps you could generate one via a classad regex on the IP address or some such. I suppose the above examples could be improved by also having non-parallel universe jobs prefer the lowest numbered MachineId machines.
BTW, if you come up with a policy that combines both parallel scheduling groups AND job ranks, be aware - I don't think that using both of these mechanisms at the same time worked until HTCondor v7.5.6.