Page History

Turn Off History

Introduction to Consumption Policies

 A Consumption Policy is a policy expression that resides on a partitionable slot classad, as advertised by the startd on an HTCondor execute node, which governs the amount of resources used by a job match against that slot.

Each kind of resource (or resource "asset") has a corresponding Consumption Policy. In a typical partitionable slot (p-slot), three resources are always defined: Cpus, Memory and Disk, which might advertise Consumption Policies as configured in this simple example: 

# enable use of consumption policies
CONSUMPTION_POLICY = True

# define a simple consumption policy
# (note, "target" refers to the scope of the candidate job classad being considered for a match)
CONSUMPTION_CPUS = target.RequestCpus
CONSUMPTION_MEMORY = target.RequestMemory
CONSUMPTION_DISK = target.RequestDisk

The HTCondor negotiator matchmaking logic is aware of a Consumption Policy (CP) detected on a p-slot. When a job matches against a p-slot with a CP, the amount of each resource dictated by its consumption policy is deducted from that p-slot. The p-slot then remains available to match with another job. In other words: Consumption Policies allow multiple jobs to be matched against a single partitionable slot during a negotiation cycle. When the HTCondor startd is allocating a claim to a new match, the same Consumption Policy expressions are also evaluated to determine the resources that are subtracted from the partitionable slot (and added to the corresponding new dynamic slot).

Motivations

Consumption Policies enable some appealing features, including support for pools with heterogeneous resource models, faster loading of partitionable slots, and more flexible utilization of accounting group quotas.

Heterogeneous Resource Models

Consumption Policies are configurable on a per-slot basis, which makes it straightforward to support multiple resource consumption models on a single pool.

For example, the following is a cpu-centric resource consumption policy: 

# cpus will generally be exhausted first:
CONSUMPTION_CPUS = target.RequestCpus
CONSUMPTION_MEMORY = quantize(target.RequestMemory, {32})
CONSUMPTION_DISK = quantize(target.RequestDisk, {128})
SLOT_WEIGHT = Cpus

Another slot might be configured with a memory-centric policy: 

# memory will generally be exhausted first:
CONSUMPTION_CPUS = 1
CONSUMPTION_MEMORY = quantize(target.RequestMemory, {1024})
CONSUMPTION_DISK = quantize(target.RequestDisk, {128})
SLOT_WEIGHT = floor(Memory / 1024)

Note that the slot weight expression is typically configured to correspond to the "most limiting" resource, and furthermore behaves as a measure of the number of potential matches remaining on the partitionable slot.

Fast Slot Loading

When the HTCondor negotiator matches a job against a partitionable slot configured with a Consumption Policy, it will deduct the resource assets (cpu, memory, etc) from that p-slot and keep it in the list. Therefore, a p-slot can be matched against multiple jobs in the same negotiation cycle. This allows p-slots to be fully loaded in a single cycle, instead of matching a single job per cycle.

(Note: the CLAIM_PARTITIONABLE_LEFTOVERS feature is an alternative approach to faster p-slot loading, operating in the scheduler as opposed to the negotiator).

Flexible Quota Utilization

The cost of matching a job against a slot is traditionally the value of the SlotWeight expression. In a scenario where the slot weights of available p-slots are greater than an accounting group's quota, the jobs in that accounting group will be starved.

However, when a p-slot with a Consumption Policy is matched then its match-cost is the change in the SlotWeight value, from before the match to after. This means that a match is only charged for the portion of the p-slot that it actually used (as measured by the SlotWeight expression), and so p-slots with many resources and possibly large SlotWeight values can generally be used by accounting groups with smaller quotas (and likewise by submitters having smaller fairshare values).

Consumption Policies and Accounting

As was mentioned above, the match cost for a job against a slot with a Consumption Policy is the change in the SlotWeight value from before the match to after. It is this match cost value that is added to the corresponding submitter's usage in the HTCondor accountant. A useful way to think about accounting with Consumption Policies is that the standard role of SlotWeight is replaced with change in SlotWeight.

Also of note: because matching with Consumption Policies takes place in the negotiator, accounting of Concurrency Limits is also implicitly handled in the standard manner.

Consumption Policy Examples

In the preceding discussion, examples of a cpu-centric and a memory-centric Consumption Policy were provided. A few other examples are listed here.

Extensible Resources

All resource assets are required to have a Consumption Policy configured. This includes Extensible Resources, if any are also present: 
# declare an extensible resource for a claim-based consumption policy
MACHINE_RESOURCE_tokens = 3
CONSUMPTION_POLICY = True
SLOT_TYPE_1 = 100%
SLOT_TYPE_1_PARTITIONABLE = True
NUM_SLOTS_TYPE_1 = 1
# always consume 1 token, and none of anything else
SLOT_TYPE_1_CONSUMPTION_TOKENS = 1
SLOT_TYPE_1_CONSUMPTION_CPUS = 0
SLOT_TYPE_1_CONSUMPTION_MEMORY = 0
SLOT_TYPE_1_CONSUMPTION_DISK = 0
# define cost in terms of available tokens for serving jobs
SLOT_TYPE_1_SLOT_WEIGHT = Tokens

emulate a static slot

This example uses a consumption policy to emulate static slot behavior 
CONSUMPTION_POLICY = True
SLOT_TYPE_1 = 100%
SLOT_TYPE_1_PARTITIONABLE = True
NUM_SLOTS_TYPE_1 = 1
# consume all resources - emulate static slot
SLOT_TYPE_1_CONSUMPTION_CPUS = TotalSlotCpus
SLOT_TYPE_1_CONSUMPTION_MEMORY = TotalSlotMemory
# Disk is unreliable -- TotalSlotDisk != Disk even on a virgin slot
SLOT_TYPE_1_CONSUMPTION_DISK = floor(0.9 * TotalSlotDisk)