CHTC Custom Collector Ads for Monitoring
The CHTC collector at cm.chtc.wisc.edu contains several custom ad types to assist monitoring. These custom ads are created by a Python script "monitorGlidein.py" that runs on cm.chtc.wisc.edu periodically via cron and lives in git repo gitolite@git.chtc.wisc.edu:puppet_data.
For example, to see instantaneous GPU job information across campus
$ condor_status -pool cm.chtc.wisc.edu -any -cons 'MyType=="GpuSubmitter"' -af:h Name Machine IdleJobs RunningJobs RunningJobsOSG HeldJobs
OsgCeScheduler Ads
These are custom scheduler ads representing HTCondor CEs deployed in OSG. For each HTCondor CE, the CE schedd is queried for aggregate information on pilot jobs (identified as jobs that have not been routed by the condor_jobrouter).
Attributes
MyType. Equals"OsgCeScheduler"Name. Identity of this CE's schedd.Machine. Host name of the CE.OsgResrouce. The abbreviated site name of the CE with_XXXsuffix appended to help ensure uniqueness (theXXXis a compressed MD5 sum on the Machine attr).IsCCStar. Boolean value; True if CE is managing a CC* site, else False.Accessible. Boolean value; if False, then the CE could not be contacted (firewall?) and thus all remaining attribute vales below will be undefined.RunningJobs. Total running pilot jobs on this CE.IdleJobs. Total idle pilot jobs on this CE.HeldJobs. Total held pilot jobs on this CE.RunningJobsGPU. Running pilot jobs on this CE that requested GPUs.IdleJobsGPU. Idle pilot jobs on this CE that requested GPUs.HeldJobsGPU. Held pilot jobs on this CE that requested GPUs.
OsgSubmitter Ads
These are custom submitter ads representing users who have jobs tagged as allowed to run in OSG (via WantGlidein=True), and submitted these jobs to a schedd requesting resources from CHTC.
Attributes
MyType. Equals"OsgSubmitter"RunningJobs. Total jobs tagged as allowed to run in OSG that are running anywhere.RunningJobsOSG. Total jobs tagged as allowed to run in OSG that are running on an OSG resource (via a pilot provisioned by CHTC's glideinwms).IdleJobs. Total jobs tagged as allowed to run in OSG that are Idle.HeldJobs. Total jobs tagged as allowed to run in OSG that are Held.Name. Identity of the submitting user.Machine. Host name of the submitting machine running the schedd.
GpuSubmitter Ads
These are custom submitter ads representing users who have jobs requesting one or more GPUs, and submitted these jobs to a schedd requesting resources from CHTC.
Attributes
MyType. Equals"GpuSubmitter"RunningJobs. Total jobs requesting GPUs that are running anywhere.RunningJobsOSG. Total jobs requesting GPUs that are running on an OSG resource (via a pilot provisioned by CHTC's glideinwms).IdleJobs. Total jobs requesting GPUs that are Idle.HeldJobs. Total jobs requesting GPUs that are Held.Name. Identity of the submitting user.Machine. Host name of the submitting machine running the schedd.
CovidSubmitter Ads
These are custom submitter ads representing users who have jobs tagged as doing COVID work (currently via OConnorCOVID=True), and submitted these jobs to a schedd requesting resources from CHTC.
Attributes
MyType. Equals"CovidSubmitter"RunningJobs. Total COVID jobs that are running anywhere.RunningJobsOSG. Total COVID jobs that are running on an OSG resource (via a pilot provisioned by CHTC's glideinwms).IdleJobs. Total COVID jobs that are Idle.HeldJobs. Total COVID jobs that are Held.Name. Identity of the submitting user.Machine. Host name of the submitting machine running the schedd.