HTCondorWiki: Adding Statistics Values

Page History

condor_utils has classes designed to make measuring and publishing statistics about the performance of code easy. The classes are defined in generic_stats.h

This consists of a set of simple templatized classes that you add as member variables to your code, or to a helper class or structure. The most commonly used of these is the stats_entry_recent class. This class keeps track of a total value, and also uses a ring buffer to keep track of recent values.

Adding statistics values to an existing collection

To add a statistics value where there is already an existing collection of values

Add a member of type stats_entry_xxx to the stats collection class. the most common type is stats_entry_recent.
Set or increment the variable in the code to be measured.
Find the ::Publish method of the collection class, it will refer to a static const array of GenericStatsPubItem. Add one or more entries in this array that refer to your new member. for stats_entry_recent type members, you will typically add a _STATS_PUB entry to publish the overall value and a _STATS_PUB_RECENT one to publish the Recent value.

This array is used by the code that publishes the collection as well as code that advances the recent buffers and sets their sizes.

Adding a statistics value when there isn't a collection

At the most basic level, use generic statistics by:

Add member variables of type stats_entry_xxx to your class or as a static or global
Set the size of the recent buffer if the type is stats_entry_recent (you can do this in the constructor)
Set or increment the variable in the code to be measured.
Periodically call the Advance method to rotate the recent buffer
call the Publish method to write the value as a ClassAd attribute.

class MyObject {
    ...
    // declare a statistics counter
    stats_entry_recent<time_t> myObjectRuntime;
    ...
};

// set statistics recent buffer to 4 items in the constructor
void MyObject::MyObject()
    : myObjectRuntime(4)
{
}

// update the counter value in one of my worker methods
void MyObject::MyWorkerMethod() {
    time_t begintime = time(NULL);

    // advance the recent buffer if enough time has passed.
    if (begintime - last_update_time)
       myObjectRuntime.AdvanceBy((begintime - last_update_time)/time_quantum);

    ... do some work

    // accumulate statistics data
    myObjectRuntime += time(NULL) - begintime;
}

// publish the counter value(s) in another worker method
void MyObject::MyPublishStatisticsMethod(ClassAd & ad) {

    // publish the overall value
    myObjectRuntime.PublishValue(ad, "ObjectRuntime");
    // publish the recent value
    myObjectRuntime.PublishRecent(ad, "RecentObjectRuntime");

}

The myObjectRuntime member will accumulate overall runtime and runtime within the recent window. In this the above example, the recent window consists of 4 intervals, with the interval advancing whenever you call the Advance or AdvanceBy methods on myObjectRuntime. You would commonly advance based on elapsed time, and there is a helper function in generic_stats.cpp generic_stats_Tick, that will advance a collection of counters on that basis.

The published ClassAd attribute data types will be the same as the templatized data type. int, double are supported in ClassAds. time_t values will be published as int.

More sophisticated counters can be built by using a class as the template parameter to stats_entry_recent. For instance stats_entry_recent<Probe> will store and publish count,min,max,average,and standard deviation for a sample value and a recent window of the values. The Probe class does count,min,max etc, and the stats_entry_recent class will manage a total and recent buffer of =Probe=s.

Creating a new statistics collection

It is common to have multiple statistics values to measure a subsystem that are intended to be published together. When this is the case is most convenient to gather then up into a container class/struct and then use the helper functions in generic_stats.cpp to Clear, Publish and Advance the collection of counters as a unit.

As a general rule a statistics collection defined by where and when the data will be gathererd rather rather than by how and where the data is published. If there is a need to publish the same value in more than one ClassAd with a different name in each add, this can be accomplished by using more than one GenericStatsPubItem table and multiple calls to the helper functions.

To take advantage of these helper functions for working with collections of predefined statistics you will.

create a structure or class to hold your counters
- use stats_entry_recent<type> for counters that should publish recent values (i.e. number of jobs that have finished)
- use stats_entry_abs<type> for counters that are instantaneous values for which you may want to publish a max value (i.e. number of shadow currently alive)
- use stats_entry_probe<type> for counters that publish min,max,avg,stdev
give your stats class methods for Init, Clear, Tick and Publish, and have those methods call the corresponding generic_stats_ functions.

class MyStats {
    time_t                  InitTime
    time_t                  Lifetime;
    time_t                  LastUpdateTime;
    time_t                  RecentLifetime;
    time_t                  RecentTickTime;

    stats_entry_recent<int> JobsSubmitted;
    stats_entry_recent<time_t> JobsTimeToStart;
    stats_entry_recent<double> JobsRunningTime;
    ... etc

    void Init();
    void Clear();
    void Tick();
    void SetRecentWindowSize(int window); // in sec
    void Publish(ClassAd & ad) const;
    void Unpublish(ClassAd & ad) const;

} MyStats;

Then in your cpp file for the MyStats class, define a static const array of GenericStatsPubItem. This array is used by the generic_stats_ helper functions.

the GENERIC_STATS_PUB_xxx macros make initializing the array with standard values easy. If you use these macros, the attribute names that are used to publish the counters will be derived from the counter variable names themselves. In the example below, MYSTATS_PUB(JobsSubmitted, AS_COUNT) results in a attribute name of MyStat_JobsSubmitted.

#define MYSTATS_PUB(name, as)         GENERIC_STATS_PUB(MyStats, "MYStat_", name, as)
#define MYSTATS_PUB_RECENT(name, as)  GENERIC_STATS_PUB_RECENT(MyStats, "MYStat_", name, as)
#define MYSTATS_PUB_TYPE(name, T, as) GENERIC_STATS_PUB_TYPE(MyStats, "MYStat_", name, as, T)


static const GenericStatsPubItem MyStatsPub[] = {
   MYSTATS_PUB_TYPE(Lifetime,       time_t,  AS_RELTIME),
   MYSTATS_PUB_TYPE(LastUpdateTime, time_t,  AS_ABSTIME),

   MYSTATS_PUB(JobsSubmitted,          AS_COUNT),
   MYSTATS_PUB_RECENT(JobsSubmitted,   AS_COUNT),
   MYSTATS_PUB(JobsTimeToStart,        AS_RELTIME),
   MYSTATS_PUB_RECENT(JobsTimeToStart, AS_RELTIME),
   MYSTATS_PUB(JobsRunningTime,        AS_RELTIME),
   MYSTATS_PUB_RECENT(JobsRunningTime, AS_RELTIME),

void MyStats::Clear()
{
   generic_stats_Clear(MyStatsPub, COUNTOF(MyStatsPub), (char*)this);
   this->InitTime = time(NULL);
   this->StatsLifetime = 0;
   this->StatsLastUpdateTime = 0;
   this->RecentStatsTickTime = 0;
   this->RecentStatsLifetime = 0;
}

void MyStats::Publish(ClassAd & ad) const
{
   generic_stats_PublishToClassAd(ad, MyStatsPub, COUNTOF(MyStatsPub), (const char *)this);
}

void MyStats::Unpublish(ClassAd & ad) const
{
   generic_stats_DeleteInClassAd(ad, MyStatsPub, COUNTOF(MyStatsPub), (const char *)this);
}

void MyStats::Tick()
{
   const int my_window  = 20*60; // seconds
   const int my_quantum = 4*60;  // in seconds

   int cAdvance = generic_stats_Tick(
      StatsTick, COUNTOF(StatsTick),
      (char*)this,
      my_window,
      my_quantum,
      this->InitTime,
      this->LastUpdateTime,
      this->RecentTickTime,
      this->Lifetime,
      this->RecentLifetime);
}

Adding statistics values at runtime

Note: this section is speculative, the code does not yet work (or work like this..)

Your statistics class can also contain an instance of stats_pool if you need to handle values that are allocated and named at runtime rather than compile time.

class MyStats : public stats_pool {
    time_t                  InitTime
    time_t                  Lifetime;
    time_t                  LastUpdateTime;
    time_t                  RecentLifetime;
    time_t                  RecentTickTime;

    void Tick();
    void NewProbe(const char * category, const char * name, int cls);
    void Sample(const char * name, int val);
    void Sample(const char * name, double val);

} MyStats;

void MyStats::NewProbe(const char * category, const char * name, int as_type)
{
    MyString attr;
    attr.sprintf("MYStat_%s_%s", category, name);
    MakeCleanAttribNameString(attr);

    int cRecent = my_window / my_quantum;
    switch (as_type)
    {
    case STATS_ENTRY_TYPE_INT32 | AS_RECENT:
       {
          stats_entry_recent<int>* probe = new stats_entry_recent<int>(cRecent);
          this->AddProbe(name, as_type, probe, attr.Value());
       }
       break;
    }
}

void MyStats::Sample(const char * name, int val)
{
    stats_entry_recent<int>* probe = this->Get<stats_entry_recent<int>*>(name);
    if (probe) probe->Add(val);
}

void MyStats::Sample(const char * name, double val)
{
    stats_entry_recent<double>* probe = this->Get<stats_entry_recent<double>*>(name);
    if (probe) probe->Add(double);
}

void MyStats::Tick()
{
   const int my_window  = 20*60; // seconds
   const int my_quantum = 4*60;  // in seconds

   int cAdvance = generic_stats_Tick(
      StatsTick, COUNTOF(StatsTick),
      (char*)this,
      my_window,
      my_quantum,
      this->InitTime,
      this->LastUpdateTime,
      this->RecentTickTime,
      this->Lifetime,
      this->RecentLifetime);

   Advance(cAdvance); // call the stats_pool Advance method.
}