HTCondor can help manage GPUs (graphics processing units) in your pool of execute nodes, making them available to jobs that can use them using an API like {link:http://www.khronos.org/opencl/ OpenCL} or {link:http://www.nvidia.com/object/cuda_home_new.html CUDA}.
-HTCondor matches execute nodes (described by ClassAds) to jobs (also described by ClassAds). The general technique to manage GPUs is:
+The techniques described on this page depend on HTCondor version 8.1.4 or later, if you are using an earlier version of HTCondor you should upgrade. If you cannot upgrade for some reason, an older, less flexible technique is
+described at HowToManageGpusInSeriesSeven
+
+HTCondor matches execute nodes (described by {quote: ClassAds}) to jobs (also described by {quote: ClassAds}). The general technique to manage GPUs is:
1: Advertise the GPU: Configure HTCondor so that execute nodes include information about available GPUs in their ClassAd.
2: Require a GPU: Jobs modify their Requirements to require a suitable GPU
3: Identify the GPU: Jobs modify their arguments or environment to learn which GPU it may use.
-This technique builds on the techniques in {wiki:HowToReserveSlotForSpecialJobs How to reserve a slot or machine for special jobs}.
-
{section: Advertising the GPU}
-A key challenge of advertising GPUs is that a GPU can only be used by one job at a time. If an execute node has multiple slots (a likely case!), you'll want to limit each GPU to only being advertised to a single slot.
-
-You have several options for advertising your GPUs. In increasing order of complexity they are:
-
-1: Static configuration
-2: Automatic configuration
-3: Dynamic advertising
-
-This progression may be a useful way to do initial setup and testing. Start with a static configuration to ensure everything works. Move to an automatic configuration to develop and test partial automation. Finally a few small changes should make it possible to turn your automatic configuration into dynamic advertising.
-
-{subsection: Static configuration}
-
-If you have a small number of nodes, or perhaps a large number
-of identical nodes, you can add static attributes manually using
-{link:http://www.cs.wisc.edu/condor/manual/v7.6/3_3Configuration.html#16198 STARTD_ATTRS} on a {link:http://www.cs.wisc.edu/condor/manual/v7.6/3_13Setting_Up.html#37111 per slot basis}. In the simplest case, it might just be:
-
-{code}
-SLOT1_HAS_GPU=TRUE
-SLOT1_GPU_DEV=0
-STARTD_ATTRS=HAS_GPU,GPU_DEV
-{endcode}
-
-This limits the GPU to only being advertised by the first slot. A job can use HAS_GPU to identify available slots with GPUs. The job can use GPU_DEV to identify which GPU device to use. (A job could use the presence of GPU_DEV to identify slots with GPUs instead of HAS_GPU, but "=HAS_GPU=" is a bit easier to read than "=(GPU_DEV=!=UNDEFINED)="
-
-If you have two GPUs, you might give the first two slots a GPU each.
-{code}
-SLOT1_HAS_GPU=TRUE
-SLOT1_GPU_DEV=0
-SLOT2_HAS_GPU=TRUE
-SLOT2_GPU_DEV=1
-STARTD_ATTRS=HAS_GPU,GPU_DEV
-{endcode}
+HTCondor contains a tool designed to assist in detecting GPUs and configuring HTCondor to advertise GPU information. This tool is
-You can also provide more information about your GPUs so that a job can distinguish between different GPUs:
+ condor_gpu_discovery
-{code}
-SLOT1_GPU_CUDA_DRV=3.20
-SLOT1_GPU_CUDA_RUN=3.20
-SLOT1_GPU_DEV=0
-SLOT1_GPU_NAME="Tesla C2050"
-SLOT1_GPU_CAPABILITY=2.0
-SLOT1_GPU_GLOBALMEM_MB=2687
-SLOT1_GPU_MULTIPROC=14
-SLOT1_GPU_NUMCORES=32
-SLOT1_GPU_CLOCK_GHZ=1.15
-SLOT1_GPU_API="CUDA"
-STARTD_ATTRS = GPU_DEV, GPU_NAME, GPU_CAPABILITY, GPU_GLOBALMEM_MB, \
- GPU_MULTIPROC, GPU_NUMCORES, GPU_CLOCK_GHZ, GPU_CUDA_DRV, \
- GPU_CUDA_RUN, GPU_MULTIPROC, GPU_NUMCORES, GPU_API
-{endcode}
+This tool will detect CUDA and OpenCL devices and output a list of GPU identifiers for all detected devices.
-(The above is from {link: https://lists.cs.wisc.edu/archive/condor-users/2011-March/msg00121.shtml Carsten Aulbert's post "RFC: Adding GPUs into Condor"}.)
+HTCondor has a general mechanism for declaring user-defined slot resources. We will use this mechanism to define a resource called 'GPUs'. The resource type name 'GPUs' is case insensitive, but you must be consistent about the plural. HTCondor considers 'GPU' to be a different resource type than 'GPUs'. We recommend the use of 'GPUs' for GPU custom resources.
+To define 'GPUs' as a custom resource simply add the following statements to the configuration on your execute node.
-{subsection:Automatic configuration}
+ MACHINE_RESOURCE_GPUs = $(LIBEXEC)/condor_gpu_discovery -properties
+ ENVIRONMENT_FOR_AssignedGPUs = CUDA_VISIBLE_DEVICES, GPU_DEVICE_ORDINAL
-You can write a program to write your
-configuration file. This is still using STARTD_ATTRS, but potentially scales
-better for mixed pools. For an extended example, see {link: https://lists.cs.wisc.edu/archive/condor-users/2011-March/msg00121.shtml Carsten Aulbert's post "RFC: Adding GPUs into Condor"} in which he does exactly this.
+The first line tells HTCondor to run the condor_gpu_discovery tool, and use it's output to define a custom resource called 'GPUs'.
-{subsection:Dynamic advertising}
+The second line tells HTCondor to publish the AssignedGPUs for a slot in the job's environment using the environment variables =CUDA_VISIBLE_DEVICES= and =GPU_DEVICE_ORDINAL=. If you know for certain that your devices will be CUDA then you can omit =GPU_DEVICE_ORDINAL= in the configuration above. If you know for certain that your devices are OpenCL only, then you can omit =CUDA_VISIBLE_DEVICES=. In addition =AssignedGPUs= will always be published into the job's environment as =_CONDOR_AssignedGPUs=, so the second line above is not strictly necessary, but it is recommended.
-One step beyond automatic configuration is dynamic configuration. Instead of a static or automated configuration, HTCondor itself can run your program and incorporate the information.
-This is {link: http://www.cs.wisc.edu/condor/manual/v7.6/4_4Hooks.html#sec:daemon-classad-hooks HTCondor's "Daemon ClassAd Hooks" functionality},
-previous known as HawkEye and HTCondor Cron. This is the route taken by the {link: http://sourceforge.net/projects/condorgpu/ condorgpu project} (Note that the condorgpu project has no affiliation with HTCondor. We have not tested or reviewed that code and cannot promise anything about it!)
+The output of the condor_gpu_discovery tool will report =DetectedGPUs= and list the GPU id of each one. GPU ids will be CUDA<n> or OCL<n> where <n> is an integer, and CUDA or OCL indicates whether the CUDA library or the OpenCL library is used to communicate with the device.
-Such a configuration might look something like this, assuming that each machine had at most two GPUs.
+The -properties argument in the command above tells condor_gpu_discovery to also list significant attributes of the device(s). These attributes will then be published in the slot ads.
+This is typical output of condor_gpu_discovery
{code}
-STARTD_CRON_JOBLIST = $(STARTD_CRON_JOBLIST) GPUINFO1
-STARTD_CRON_GPUINFO1_MODE = OneShot
-STARTD_CRON_GPUINFO1_RECONFIG_RERUN = FALSE
-STARTD_CRON_GPUINFO1_PREFIX = GPU_
-STARTD_CRON_GPUINFO1_EXECUTABLE = $(MODULES)/get-gpu-info
-# which device should get-gpu-info probe?
-STARTD_CRON_GPUINFO1_PARAM0 = 0
-STARTD_CRON_GPUINFO1_SLOTS = 1
-
-STARTD_CRON_JOBLIST = $(STARTD_CRON_JOBLIST) GPUINFO2
-STARTD_CRON_GPUINFO2_MODE = OneShot
-STARTD_CRON_GPUINFO2_RECONFIG_RERUN = FALSE
-STARTD_CRON_GPUINFO2_PREFIX = GPU_
-STARTD_CRON_GPUINFO2_EXECUTABLE = $(MODULES)/get-gpu-info
-# which device should get-gpu-info probe?
-STARTD_CRON_GPUINFO2_PARAM0 = 1
-STARTD_CRON_GPUINFO2_SLOTS = 2
+> condor_gpu_discovery -properties
+DetectedGPUs="CUDA0, CUDA1, CUDA2, CUDA3"
+CUDACapability=3.0
+CUDADeviceName="GeForce GTX 690"
+CUDADriverVersion=5.50
+CUDAECCEnabled=false
+CUDAGlobalMemory=2048
+CUDARuntimeVersion=5.0
{endcode}
-$(MODULES)/get-gpu-info will be invoked twice, once for each of the two possible GPUs. (You can support more by copying the above entries and increasing the integers. #2196, if implemented, may allow for a simpler configuration.) get-gpu-info will be passed the device ID to probe (0 or 1). The output should be a ClassAd; entries will have GPU_ prepended, then they will be added to to slot ClassAds for slots 1 and 2.
-
-get-gpu-info would write output to its standard output that looked something like:
+This output indicates that 4 GPUs were detected, all of which have the same properties. If you are using a static slot configuration, and you wish to
+control how many GPUs are assigned to each slot, use the =SLOT_TYPE_<n>= configuration syntax to specify =Gpus=, the same as you would for =Cpus= or =Memory=. If you don't specify, slots default to =GPUS=auto= which will assign
+GPUs proportionally to slots until there are no more GPUs to assign, then it will simply assign 0 GPUs to the remaining slots. So a machine with =NUM_CPUS=8= and =DetectedGPUs="CUDA0, CUDA1, CUDA2, CUDA3"= will assign 1 GPUs each to the first 4 slots, and no GPUs to the remaining slots. Slots with GPUs assigned with include the following attributes.
{code}
-CUDA_DRV=3.20
-CUDA_RUN=3.20
-DEV=0
-NAME="Tesla C2050"
-CAPABILITY=2.0
-GLOBALMEM_MB=2687
-MULTIPROC=14
-NUMCORES=32
-CLOCK_GHZ=1.15
-API="CUDA"
+Cpus=1
+GPUs=1
+TotalCpus=8
+TotalGPUs=4
+TotalSlotCpus=1
+TotalSlotGPUs=1
+CUDACapability=3.0
+CUDADeviceName="GeForce GTX 690"
+CUDADriverVersion=5.50
+CUDAECCEnabled=false
+CUDAGlobalMemory=2048
+CUDARuntimeVersion=5.0
{endcode}
-{link: https://lists.cs.wisc.edu/archive/condor-users/2011-March/msg00121.shtml Carsten Aulbert's post "RFC: Adding GPUs into HTCondor"} includes a program that might make a good starting point for writing output like the above.
+
+If you are using a Partitionable slot, the by default the Partitionable slot will be assigned both GPUs. Dynamic slots created from this partitionable slot will be assigned GPUs when the job requests them.
{section: Require a GPU}
@@ -124,37 +73,26 @@
User jobs that require a GPU must specify this requirement. In a job's submit file, it might do something as simple as
{code}
-Requirements=HAS_GPU
+Request_GPUs = 1
{endcode}
or as complex as
{code}
-Requirements=HAS_GPU \
- && (GPU_API == "CUDA") \
- && (GPU_NUM_CORES >= 16) \
- && regexp("Tesla", NAME, "i")
+Request_GPUs = 2
+Requirements = CUDARuntimeVersion >= 5.5 \
+ && (CUDACapability >= 3.0) \
+ && (CUDAGlobalMemoryMb >= 1500)
{endcode}
-specifying that the job requires the CUDA GPU API (as opposed to OpenCL or another), that it wants a GPU with at least 16 cores, and it wants a GPU with a name of "Tesla".
-
-
+specifying that the job requires a CUDA GPU with at least 1500 Mb of memory, the CUDA runtime version 5.5 or later, and a CUDA Capability of 3.0 or greater.
{section: Identify the GPU}
-Once a job matches to a given slot, it needs to know which GPU to use, if multiple are present. Assuming the slot advertised the information, you can access it through the job's arguments or the environment using The $$() syntax. For example, if your job takes an argument "--device=X" where X is the device to use, you might do something like
+Once a job matches to a given slot, it needs to know which GPUs to use, if multiple are present. GPUs that the job are permitted to use are specified in the slot {quote: ClassAd} as =AssignedGPUs=. They are also published into the job's environment as =_CONDOR_AssignedGPUs=, and (if the configuration at the top is used) as =CUDA_VISIBLE_DEVICES= and =GPU_DEVICE_ORDINAL=. You can pass Assigned GPUs to your jobs arguments using the $$() syntax. For example, if your job takes an argument "--device=X" where X is the device to use, you might do something like
{code}
-arguments = "--device=$$(GPU_DEV)"
+arguments = "--device=$$(AssignedGPUs)"
{endcode}
-Or your job might look to the environment variable GPU_DEVICE_ID:
-
-{code}
-environment = "GPU_DEVICE_ID=$$(GPU_DEV)"
-{endcode}
-
-
-{section: The Future}
-
-The HTCondor team is working on various improvements in how HTCondor can manage GPUs. We're interested in how you are currently using GPUs in your cluster and how you plan on using them. If you have thoughts or questions, you can post to the public {link: http://www.cs.wisc.edu/condor/mail-lists/ condor-users mailing list}, or {link: http://www.cs.wisc.edu/condor/condor-support/contact us directly}.
+Or your job might look to the environment variable =_CONDOR_AssignedGPUs=, or =CUDA_VISIBLE_DEVICES=
