{section: How to configure priorities/quotas for groups of users}
-Known to work with HTCondor version: 7.0
-
-By default, HTCondor provides fair sharing between individual users by keeping track of usage and adjusting their relative user priorities in the pool. Frequently, however, there is a need to allocate resources at a higher level. Suppose you have a single HTCondor pool shared by several groups of users. Your goal is to configure HTCondor so that each group gets its fair share of the computing resources and within each group, each user gets a fair share relative to other members of the group. What is "fair" depends on circumstances, such as whether some groups own a larger share of the machines than others and whether some groups have a different pattern of usage such as occasional bursts of computation verses steady demand.
+By default, HTCondor provides fair sharing between individual users by keeping track of usage and adjusting their relative user priorities. Frequently, however, there is a need to allocate resources at a more abstract level. Suppose you have a single HTCondor pool shared by several sets of users. Your goal is to configure HTCondor so that each group (set) gets its fair share of the computing resources and within each group, each user gets a fair share relative to other members of the group. What is defined to be fair depends on circumstances, such as whether some groups own a larger share of the machines than others and whether some groups have a different pattern of usage such as occasional bursts of computation verses steady demand.
The following recipes can be adapted to a variety of such situations.
-{subsection: How to divide the machines by startd RANK}
+{subsection: How to divide the machines by the machine's RANK}
-You can give a group of users higher priority on a specific set of machines by using the startd RANK expression. Example:
+You can give a group of users higher priority on a specific set of machines by configuring the =RANK= expression. Example:
{code}
# This machine belongs to the "biology" group.
@@ -19,13 +17,13 @@
Rank = TARGET.Group =?= MY.MachineOwner
{endcode}
-The above example requires that jobs be submitted with an additional custom attribute "Group" that declares what group they belong to. See {wiki: HowToInsertCustomClassAd How to insert custom ClassAd attributes into a job ad} for different ways of doing that. One way is simply for the user to do it explicitly in the submit file:
+The above example requires that jobs be submitted with an additional custom attribute called ="Group"= in this example, that declares what group they belong to. See {wiki: HowToInsertCustomClassAd How to insert custom ClassAd attributes into a job ad} for different ways of doing that. The user may set this explicitly in the job's submit description file with
{code}
+Group = "biology"
{endcode}
-If group membership is not likely to change frequently, or you really don't want users to have to declare their group membership, you could configure the startd RANK expression to look at the built-in TARGET.User attribute rather than relying on the custom attribute TARGET.Group. Example:
+If group membership is not likely to change frequently, or you really don't want users to have to declare their group membership, you could configure the machine =RANK= expression to look at the built-in =TARGET.User= attribute, rather than relying on the custom attribute =TARGET.Group=. An example configuration that implements this:
{code}
MachineOwners = "user1@biology.wisc.edu user2@biology.wisc.edu"
@@ -34,10 +32,9 @@
RANK = stringListMember(TARGET.User,MY.MachineOwners)
{endcode}
-Since RANK is an arbitrary {quote: ClassAd} expression, you can customize the policy in a number of ways. For example, there could be a group with second priority on the machines. Or you could specify that some types of jobs (identified by some other {quote: ClassAd} attribute in the job) have higher priorities than others. You just need to write the expression so that it produces a higher number for higher priority jobs.
+Since =RANK= is an arbitrary ClassAd expression, you can customize the policy in a variety of ways. For example, there could be a group with second priority on the machines. Or you could specify that some types of jobs (identified by some other ClassAd attribute in the job) have higher priorities than others. In that case, write the expression so that it produces a higher number for higher priority jobs.
-The down side of RANK is that it involves preemption. RANK only comes into play
-when there is an existing job on a machine and the negotiator is considering whether a new job should preempt it. You can control how quickly the preemption happens in order for the new job to replace the lower priority job using {quote: MaxJobRetirementTime} as described in {wiki: HowToDisablePreemption How to disable preemption}. By default, the preemption will happen immediately. This is most appropriate in HTCondor pools where groups own specific machines and want guaranteed access to them whenever they need them.
+The down side of =RANK= is that it involves preemption. =RANK= only comes into play when there is an existing job on a machine and the negotiator is considering whether a new job should preempt it. You can control how quickly the preemption happens in order for the new job to replace the lower priority job using =MaxJobRetirementTime= as described in {wiki: HowToDisablePreemption How to disable preemption}. By default, the preemption will happen immediately. This is most appropriate in HTCondor pools where groups own specific machines and want guaranteed access to them whenever they need them.
Given a choice of two machines to run a job on, it is a good idea to steer jobs towards machines that rank them higher so they stand less of a chance of being preempted in the future. Here is an example configuration that preferentially runs jobs where they are most highly ranked and secondarily prefers to run jobs on idle machines rather than claimed machines:
@@ -45,56 +42,8 @@
NEGOTIATOR_PRE_JOB_RANK = 10 * (MY.RANK) + 1 * (RemoteOwner =?= UNDEFINED)
{endcode}
-Note that preemption by startd RANK trumps considerations of user priority in the pool. For example, if user A with a high (bad) user priority is competing with another user B with a low (good) user priority, user B, with the better user priority, will be able to claim more idle machines, but user A can then preempt user B if the startd RANK is higher. The relative user priorities therefore only matter when the startd RANKs are equal or when vying for unclaimed machines.
-
-{subsection: How to use group accounting to implement pool shares}
-
-Whereas startd RANK can be used to give groups preemptive priority on specific machines, HTCondor's group accounting system can be used to give groups a share of a pool that is not tied to specific machines. The HTCondor manual section on {link: http://www.cs.wisc.edu/condor/manual/v7.0/3_4User_Priorities.html#SECTION00446000000000000000" Group Accounting} covers this topic.
-
-Basically, there are two options:
-
-*: 1. Identities of individuals can be ignored for purposes of accounting and sharing the pool. Only the group the user belongs to is relevant, and each group has its own priority, which can be weighted as appropriate, and which can be automatically adjusted by HTCondor's fair sharing algorithm.
-
-*: 2. Individual identities are used when accounting and allocating resources, but each group is assigned a quota of batch slots. If the group is currently using fewer slots than its quota, then its members get preferential treatment when vying for resources.
-
-
-Quick examples of these two options are provided below.
-
-{subsection: How to use group membership exclusively for sharing the pool}
-
-Users must simply declare their membership in a group. This is done in the submit file:
-
-{code}
-+AccountingGroup = "group_physics"
-{endcode}
+Note that preemption by =RANK= trumps considerations of user priority within the pool. For example, if user A with a high (bad) user priority is competing with another user B with a low (good) user priority, user B, with the better user priority, will be able to claim more idle machines, but user A can then preempt user B if the =RANK= is higher. The relative user priorities therefore only matter when the =RANK= values are equal or when vying for unclaimed machines.
-The administrator can then adjust the priority factor for the various groups using condor_userprio. The following example sets the biology group's priority factor to be 4x that of physics, which means the physics group should get twice as big a share of the pool at steady state (for an explanation, see {wiki: HowToConfigureFairShare How to Configure Fair Share}).
+{subsection: Group accounting to implement pool shares}
-{code}
-condor_userprio -setfactor 2.0 group_physics@wisc.edu
-condor_userprio -setfactor 8.0 group_biology@wisc.edu
-{endcode}
-
-{subsection: How to have group quotas and individual priorities for sharing the pool}
-
-In the negotiator configuration:
-
-{code}
-# define the groups in your pool
-GROUP_NAMES = group_physics, group_biology, group_chemistry
-
-# specify what share of the pool (# of batch slots) each group should have
-GROUP_QUOTA_group_physics = 100
-GROUP_QUOTA_group_biology = 75
-GROUP_QUOTA_group_chemistry = 80
-
-# specify that once a group has used its full quota, users from within
-# the group can still vie for additional resources as individuals
-GROUP_AUTOREGROUP = True
-{endcode}
-
-The submit file of the job needs to specify which accounting group and user within the group it belongs to:
-
-{code}
-+AccountingGroup = "group_physics.newton"
-{endcode}
+Whereas =RANK= can be used to give groups preemptive priority on specific machines, HTCondor's group accounting system can be used to give groups a share of a pool that is not tied to specific machines. The HTCondor manual sections on Group Accounting and Accounting Groups with Hierarchical Group Quotas covers this topic. Please locate the manual sections for descriptions and implementation examples to follow.
