{section: CHTC Custom Collector Ads for Monitoring}

The CHTC collector at cm.chtc.wisc.edu contains several custom ad types to assist monitoring.  These custom ads are created by a Python script "monitorGlidein.py" that runs on cm.chtc.wisc.edu periodically via cron and lives in git repo gitolite@git.chtc.wisc.edu:puppet_data.

For example, to see instantaneous GPU job information across campus

   $ condor_status -pool cm.chtc.wisc.edu -any -cons 'MyType=="GpuSubmitter"' -af:h Name Machine IdleJobs RunningJobs RunningJobsOSG HeldJobs

{subsection: OsgCeScheduler Ads}

These are custom scheduler ads representing HTCondor CEs deployed in OSG. For each HTCondor CE, the CE schedd is queried for aggregate information on pilot jobs (identified as jobs that have not been routed by the condor_jobrouter).

{subsubsection: Attributes}

*: =MyType=. Equals ="OsgCeScheduler"=
*: =Name=. Identity of this CE's schedd.
*: =Machine=. Host name of the CE.
*: =OsgResrouce=. The abbreviated site name of the CE with =_XXX= suffix appended to help ensure uniqueness (the =XXX= is a compressed MD5 sum on the Machine attr).
*: =IsCCStar=. Boolean value; True if CE is managing a CC* site, else False.
*: =Accessible=. Boolean value; if False, then the CE could not be contacted (firewall?) and thus all remaining attribute vales below will be undefined.
*: =RunningJobs=. Total running pilot jobs on this CE.
*: =IdleJobs=. Total idle pilot jobs on this CE.
*: =HeldJobs=. Total held pilot jobs on this CE.
*: =RunningJobsGPU=. Running pilot jobs on this CE that requested GPUs.
*: =IdleJobsGPU=. Idle pilot jobs on this CE that requested GPUs.
*: =HeldJobsGPU=. Held pilot jobs on this CE that requested GPUs.

{subsection: OsgSubmitter Ads}

These are custom submitter ads representing users who have jobs tagged as allowed to run in OSG (via =WantGlidein=True=), and submitted these jobs to a schedd requesting resources from CHTC.

{subsubsection: Attributes}

*: =MyType=. Equals ="OsgSubmitter"=
*: =RunningJobs=. Total jobs tagged as allowed to run in OSG that are running anywhere.
*: =RunningJobsOSG=. Total jobs tagged as allowed to run in OSG that are running on an OSG resource (via a pilot provisioned by CHTC's glideinwms).
*: =IdleJobs=. Total jobs tagged as allowed to run in OSG that are Idle.
*: =HeldJobs=. Total jobs tagged as allowed to run in OSG that are Held.
*: =Name=. Identity of the submitting user.
*: =Machine=. Host name of the submitting machine running the schedd.

{subsection: GpuSubmitter Ads}

These are custom submitter ads representing users who have jobs requesting one or more GPUs, and submitted these jobs to a schedd requesting resources from CHTC.

{subsubsection: Attributes}

*: =MyType=. Equals ="GpuSubmitter"=
*: =RunningJobs=. Total jobs requesting GPUs that are running anywhere.
*: =RunningJobsOSG=. Total jobs requesting GPUs that are running on an OSG resource (via a pilot provisioned by CHTC's glideinwms).
*: =IdleJobs=. Total jobs requesting GPUs that are Idle.
*: =HeldJobs=. Total jobs requesting GPUs that are Held.
*: =Name=. Identity of the submitting user.
*: =Machine=. Host name of the submitting machine running the schedd.


{subsection: CovidSubmitter Ads}

These are custom submitter ads representing users who have jobs tagged as doing COVID work (currently via =OConnorCOVID=True=), and submitted these jobs to a schedd requesting resources from CHTC.

{subsubsection: Attributes}

*: =MyType=. Equals ="CovidSubmitter"=
*: =RunningJobs=. Total COVID jobs that are running anywhere.
*: =RunningJobsOSG=. Total COVID jobs that are running on an OSG resource (via a pilot provisioned by CHTC's glideinwms).
*: =IdleJobs=. Total COVID jobs that are Idle.
*: =HeldJobs=. Total COVID jobs that are Held.
*: =Name=. Identity of the submitting user.
*: =Machine=. Host name of the submitting machine running the schedd.