HTCondorWiki: How To Manage Gpus

     ENVIRONMENT_FOR_AssignedGPUs = GPU_DEVICE_ORDINAL=/(CUDA|OCL)// CUDA_VISIBLE_DEVICES
 
 
-=MACHINE_RESOURCE_INVENTORY_GPUs= tells HTCondor to run the _condor_gpu_discovery_ tool, and use its output to define a custom resource called =GPUs=.
+=MACHINE_RESOURCE_INVENTORY_GPUs= tells HTCondor to run the _condor_gpu_discovery_ tool, and use its output to define a custom resource with the resource tag =GPUs=.
 
-=ENVIRONMENT_FOR_AssignedGPUs= tells HTCondor to publish the value of machine ClassAd attribute =AssignedGPUs= for a slot in the job's environment using the environment variables =CUDA_VISIBLE_DEVICES= and =GPU_DEVICE_ORDINAL=. If you know for certain that your devices will be CUDA, then you can omit =GPU_DEVICE_ORDINAL= in the configuration above.  If you know for certain that your devices are OpenCL only, then you can omit =CUDA_VISIBLE_DEVICES=. In addition, =AssignedGPUs= will always be published into the job's environment as =_CONDOR_AssignedGPUs=, so the second line above is not strictly necessary, but it is recommended.
+=ENVIRONMENT_FOR_AssignedGPUs= tells HTCondor to publish the value of machine ClassAd attribute =AssignedGPUs= for a slot in the job's environment using the environment variables =GPU_DEVICE_ORDINAL= and =CUDA_VISIBLE_DEVICES=. In addition, =AssignedGPUs= will always be published into the job's environment as =_CONDOR_AssignedGPUs=.
 
-The output of the _condor_gpu_discovery_ tool will report =DetectedGPUs= and list the GPU id of each one.  GPU ids will be CUDA<n> or OCL<n>, where <n> is an integer, and CUDA or OCL indicates whether the CUDA library or the OpenCL library is used to communicate with the device.
+The output of the _condor_gpu_discovery_ tool reports =DetectedGPUs= and lists the GPU id of each one.  GPU ids will be CUDA<n> or OCL<n>, where <n> is an integer, and CUDA or OCL indicates whether the CUDA library or the OpenCL library is used to communicate with the device.
 
 The =-properties= argument in the _condor_gpu_discovery_ command tells it to also list significant attributes of the device(s). These attributes will then be published in each slot ClassAd.
 
-Here is typical output of _condor_gpu_discovery_
+Here is typical output of _condor_gpu_discovery_:
 {code}
 > condor_gpu_discovery -properties
 DetectedGPUs="CUDA0, CUDA1, CUDA2, CUDA3"
@@ -55,7 +55,9 @@
 CUDARuntimeVersion=5.0
 {endcode}
 
-This output indicates that 4 GPUs were detected, all of which have the same properties. If using a static slot configuration, to
+This output indicates that 4 GPUs were detected, all of which have the same properties.
+{subsection: Extra configuration}
+If using a static slot configuration, to
 control how many GPUs are assigned to each slot, use the =SLOT_TYPE_<n>= configuration syntax to specify =Gpus=, the same as would be done for =Cpus= or =Memory=. If not specified, slots default to =GPUS=auto=, which will assign
 GPUs proportionally to slots until there are no more GPUs to assign, and then it will assign 0 GPUs to the remaining slots. So a machine with =NUM_CPUS=8= and =DetectedGPUs="CUDA0, CUDA1, CUDA2, CUDA3"= will assign 1 GPUs each to the first 4 slots, and no GPUs to the remaining slots. Slot ClassAds with GPUs assigned will include the following attributes:
 
@@ -83,14 +85,14 @@
 User jobs that require a GPU must specify this requirement.  In a job's submit description file, the  simple request is
 
 {code}
-Request_GPUs = 1
+request_GPUs = 1
 {endcode}
 
 A more complex request, such as:
 
 {code}
-Request_GPUs = 2
-Requirements = CUDARuntimeVersion >= 5.5 \
+request_GPUs = 2
+requirements = CUDARuntimeVersion >= 5.5 \
     && (CUDACapability >= 3.0) \
     && (CUDAGlobalMemoryMb >= 1500)
 {endcode}
@@ -99,7 +101,7 @@
 
 {section: 3. Identify the GPU}
 
-Once a job matches to a given slot, it needs to know which GPUs to use, if multiple are present.  GPUs that the job are permitted to use are specified as defined values for the slot ClassAd attribute =AssignedGPUs=.  They are also published into the job's environment with variable =_CONDOR_AssignedGPUs=. In addition, if the configuration is defined with =ENVIRONMENT_FOR_AssignedGPUs= set, environment variables =CUDA_VISIBLE_DEVICES= and =GPU_DEVICE_ORDINAL= are published. The =AssignedGPUs= attribute value can be passed as job arguments using the $$() substitution macro syntax.  For example, if the job takes an argument "--device=X" where X is the device to use, specify this in the submit description file with
+Once a job matches to a given slot, it needs to know which GPUs to use, if multiple are present.  GPUs that the job are permitted to use are specified as defined values for the slot ClassAd attribute =AssignedGPUs=.  They are also published into the job's environment with variable =_CONDOR_AssignedGPUs=. In addition, if the configuration is defined with =ENVIRONMENT_FOR_AssignedGPUs= set, environment variables =CUDA_VISIBLE_DEVICES= and =GPU_DEVICE_ORDINAL= are published. The =AssignedGPUs= attribute value can be accessed if passed to the job as arguments using the $$() substitution macro syntax.  For example, if the job takes an argument "--device=X" where X is the device to use, specify this in the submit description file with
 
 {code}
 arguments = "--device=$$(AssignedGPUs)"