Page History

Turn Off History

Introduction to Consumption Policies

 A Consumption Policy is a policy expression that resides on a partitionable slot classad, as advertised by the startd on an HTCondor execute node, which governs the amount of resources used by a job match against that slot.

Each kind of resource (or resource "asset") has a corresponding Consumption Policy. In a typical partitionable slot (p-slot), three resources are always defined: Cpus, Memory and Disk, which might advertise Consumption Policies as configured in this simple example: 

# enable use of consumption policies
CONSUMPTION_POLICY = True

# define a simple consumption policy
# (note, "target" refers to the scope of the candidate job classad being considered for a match)
CONSUMPTION_CPUS = target.RequestCpus
CONSUMPTION_MEMORY = target.RequestMemory
CONSUMPTION_DISK = target.RequestDisk

The HTCondor negotiator matchmaking logic is aware of a Consumption Policy (CP) detected on a p-slot. When a job matches against a p-slot with a CP, the amount of each resource dictated by its consumption policy is deducted from that p-slot. The p-slot then remains available to match with another job. In other words: Consumption Policies allow multiple jobs to be matched against a single partitionable slot during a negotiation cycle. When the HTCondor startd is allocating a claim to a new match, the same Consumption Policy expressions are also evaluated to determine the resources that are subtracted from the partitionable slot (and added to the corresponding new dynamic slot).

Motivations

Consumption Policies enable some appealing features, including support for pools with heterogeneous resource models, faster loading of partitionable slots, and more flexible utilization of accounting group quotas.

Heterogeneous Resource Models

Consumption Policies are configurable on a per-slot basis, which makes it straightforward to support multiple resource consumption models on a single pool.

For example, the following is a cpu-centric resource consumption policy: 

# cpus will generally be exhausted first
CONSUMPTION_CPUS = target.RequestCpus
CONSUMPTION_MEMORY = quantize(target.RequestMemory, {32})
CONSUMPTION_DISK = quantize(target.RequestDisk, {128})
SLOT_WEIGHT = Cpus

Another slot might be configured with a memory-centric policy: 

# memory will generally be exhausted first
CONSUMPTION_CPUS = 1
CONSUMPTION_MEMORY = quantize(target.RequestMemory, {1024})
CONSUMPTION_DISK = quantize(target.RequestDisk, {128})
SLOT_WEIGHT = floor(Memory / 1024)

Note that the slot weight expression is typically configured to correspond to the "most limiting" resource, and furthermore behaves as a measure of the number of matches remaining on the partitionable slot.

Fast Slot Loading

Flexible Quota Utilization

Consumption Policies and Accounting

Examples