Is the idea that you want to _require_ all the nodes of a 4-way or 8-way parallel universe job to run on the same physical machine (i.e. to be certain a parallel job never has to communicate over the network) ? If so, one easy option may be to simply create a unique PARALLEL_SCHEDULING_GROUP for each machine, e.g. append in condor_config: {code} ParallelSchedulingGroup = "$(FULL_HOSTNAME)" STARTD_ATTRS = $(STARTD_ATTRS), ParallelSchedulingGroup {endcode} and then add to your submit file: {code} +WantParallelSchedulingGroups = True {endcode} For more info on parallel scheduling groups, see section 3.12.9.4, HTCondor's Dedicated Scheduling section in the HTCondor Manual. If you don't wish to require packing all nodes of a parallel job onto the same physical machine, you could tell HTCondor to attempt to use the least number of physical machines to run a parallel job as follows: {code} ## The NEGOTIATOR_PRE_JOB_RANK expression overrides all other ranks ## that are used to pick a match from the set of possibilities. ## The following expression matches jobs to unclaimed slots ## whenever possible, regardless of the job-supplied rank. ## For all slots, prefer to give out the highest MachineId ## available to parallel universe jobs in order to try to pack parallel ## jobs onto the the minimum number of physical machines. NEGOTIATOR_PRE_JOB_RANK =(1000000*RemoteOwner =?= UNDEFINED) \ + ( (TARGET.JobUniverse=?=11)*MachineId ) ## Now, if the job is a parallel universe job, tell condor_submit to ## append a job rank expression in the job classads that prefers to use ## the highest MachineId available. Note that for parallel universe ## jobs it is not enough to set NEGOTIATOR ranking alone to use ## the highest MachineId available, we must also put it into the job ## classad itself because the dedicated scheduler does its own ## re-sorting of jobs to machine mappings after it gets its matches ## from the negotiator. APPEND_RANK = (JobUniverse=?=11)*MachineId {endcode} The above example assumes you have a global unique ordering of your physical machines as an integer in machine ad attribute {quote: MachineId}. If you cannot easily add such a global ordering, perhaps you could generate one via a classad regex on the IP address or some such. I suppose the above examples could be improved by also having non-parallel universe jobs prefer the lowest numbered {quote: MachineId} machines. BTW, if you come up with a policy that combines both parallel scheduling groups AND job ranks, be aware - I don't think that using both of these mechanisms at the same time worked until HTCondor v7.5.6.