{section: CHTC Custom Collector Ads for Monitoring}

The CHTC collector at cm.chtc.wisc.edu contains several custom ad types to assist monitoring.  These custom ads are created by a Python script "monitorGlidein.py" that runs on cm.chtc.wisc.edu periodically via cron and lives in git repo gitolite@git.chtc.wisc.edu:puppet_data.

{subsection: OsgCeScheduler Ads}

These are custom scheduler ads representing HTCondor CEs deployed in OSG. For each HTCondor CE, the CE schedd is queried for aggregate information on pilot jobs (identified as jobs that have not been routed by the condor_jobrouter).

{subsubsection: Attributes}

*: =MyType=. Equals ="OsgCeScheduler"=
*: =Name=. Identity of this CE's schedd.
*: =Machine=. Host name of the CE.
*: =Accessible=. Boolean value; if False, then the CE could not be contacted (firewall?) and thus all remaining attribute vales below will be zero.
*: =RunningJobs=. Total running pilot jobs on this CE.
*: =IdleJobs=. Total idle pilot jobs on this CE.
*: =HeldJobs=. Total held pilot jobs on this CE.
*: =RunningJobsGPU=. Running pilot jobs on this CE that requested GPUs.
*: =IdleJobsGPU=. Idle pilot jobs on this CE that requested GPUs.
*: =HeldJobsGPU=. Held pilot jobs on this CE that requested GPUs.

{subsection: OsgSubmitter Ads}

These are custom submitter ads representing users who have jobs tagged as allowed to run in OSG (via =WantGlidein=True=), and submitted these jobs to a schedd requesting resources from CHTC.

{subsubsection: Attributes}

*: =MyType=. Equals ="OsgSubmitter"=
*: =RunningJobs=. Total jobs tagged as allowed to run in OSG that are running anywhere.
*: =RunningJobsOSG=. Total jobs tagged as allowed to run in OSG that are running on an OSG resource (via a pilot provisioned by CHTC's glideinwms).
*: =IdleJobs=. Total jobs tagged as allowed to run in OSG that are Idle.
*: =HeldJobs=. Total jobs tagged as allowed to run in OSG that are Held.
*: =Name=. Identity of the submitting user.
*: =Machine=. Host name of the submitting machine running the schedd.

{subsection: GpuSubmitter Ads}

These are custom submitter ads representing users who have jobs requesting one or more GPUs, and submitted these jobs to a schedd requesting resources from CHTC.

{subsubsection: Attributes}

*: =MyType=. Equals ="GpuSubmitter"=
*: =RunningJobs=. Total jobs requesting GPUs that are running anywhere.
*: =RunningJobsOSG=. Total jobs requesting GPUs that are running on an OSG resource (via a pilot provisioned by CHTC's glideinwms).
*: =IdleJobs=. Total jobs requesting GPUs that are Idle.
*: =HeldJobs=. Total jobs requesting GPUs that are Held.
*: =Name=. Identity of the submitting user.
*: =Machine=. Host name of the submitting machine running the schedd.


{subsection: CovidSubmitter Ads}

These are custom submitter ads representing users who have jobs tagged as doing COVID work (currently via =OConnorCOVID=True=), and submitted these jobs to a schedd requesting resources from CHTC.

{subsubsection: Attributes}

*: =MyType=. Equals ="CovidSubmitter"=
*: =RunningJobs=. Total COVID jobs that are running anywhere.
*: =RunningJobsOSG=. Total COVID jobs that are running on an OSG resource (via a pilot provisioned by CHTC's glideinwms).
*: =IdleJobs=. Total COVID jobs that are Idle.
*: =HeldJobs=. Total COVID jobs that are Held.
*: =Name=. Identity of the submitting user.
*: =Machine=. Host name of the submitting machine running the schedd.