HTCondor has a general mechanism for declaring user-defined slot resources. GPUs are a user-defined slot resource, so this same mechanism is used to define a resource. Use of the metaknob always uses the resource type name =GPUs=. This resource type name is case insensitive, but all characters within the name are significant,
so be consistent.
-Define =GPUs= as a custom resource by adding the following definitions to the configuration of the execute node.
+{subsection:What the metaknob configures}
+The =feature GPUs= defines a custom resource with this configuration of the execute node:
MACHINE_RESOURCE_INVENTORY_GPUs = $(LIBEXEC)/condor_gpu_discovery -properties
- ENVIRONMENT_FOR_AssignedGPUs = CUDA_VISIBLE_DEVICES, GPU_DEVICE_ORDINAL
+ ENVIRONMENT_FOR_AssignedGPUs = GPU_DEVICE_ORDINAL=/(CUDA|OCL)// CUDA_VISIBLE_DEVICES
=MACHINE_RESOURCE_INVENTORY_GPUs= tells HTCondor to run the _condor_gpu_discovery_ tool, and use its output to define a custom resource called =GPUs=.
@@ -96,7 +97,7 @@
specifies that the job requires a CUDA GPU with at least 1500 Mb of memory, the CUDA runtime version 5.5 or later, and a CUDA Capability of 3.0 or greater.
-{section: Identify the GPU}
+{section: 3. Identify the GPU}
Once a job matches to a given slot, it needs to know which GPUs to use, if multiple are present. GPUs that the job are permitted to use are specified as defined values for the slot ClassAd attribute =AssignedGPUs=. They are also published into the job's environment with variable =_CONDOR_AssignedGPUs=. In addition, if the configuration is defined with =ENVIRONMENT_FOR_AssignedGPUs= set, environment variables =CUDA_VISIBLE_DEVICES= and =GPU_DEVICE_ORDINAL= are published. The =AssignedGPUs= attribute value can be passed as job arguments using the $$() substitution macro syntax. For example, if the job takes an argument "--device=X" where X is the device to use, specify this in the submit description file with
