condor_transform_ads language

Transform rules files consist of lines containing key=value pairs or transform commands such as SET, RENAME, etc. Transform commands execute as they are read and can make use of values set up until that point using the $(key) macro substitution commands that HTCondor configuration files and condor_submit files use.

Most constructs that work in these files will also work in rules files such as if/else. Macro substitution will fetch attributes of the ClassAd to be transformed when $(MY.attr) is used.

The transform commands are:

SET <attr> <expr> Set <attr> to <expr>
EVALSET <attr> <expr> Evaluate <expr> and then set <attr> to the result
DEFAULT <attr> <expr> Set <attr> to <expr> if <attr> is undefined or missing
COPY <attr> <newattr> Copy the value of <attr> to <newattr>
COPY /<regex>/ <newattrs> Copy the values of attributes whose names match <regex> to <newattrs>
RENAME <attr> <newattr> Rename <attr> to <newattr>
RENAME /<regex>/ <newattrs> Rename attributes matching <regex> to <newattrs>
DELETE <attr> <newattr> Delete <attr>
DELETE /<regex>/ Delete attributes matching <regex>
EVALMACRO <key> <expr> Evaluate <expr> and then insert it as a transform macro value

In the above commands <attr> must be a valid attribute name and <expr> a valid ClassAd expression or literal. The various $() macros will be expanded in <expr>, <newattr> or <newattrs> before they are parsed as ClassAd expressions or attribute names.

When a COPY, RENAME, or DELETE with <regex> is used, regex capture groups are substituted in <newattrs> after $() expansion. \0 will expand to the entire match, \1 to the first capture, etc.

Optionally, a transform rule set can end with an iteration command

TRANSFORM [<N>] [<vars>] [in <list> | from <file> | matching <pattern>]

A TRANSFORM command must be the last command in the rules file. It takes the same options as the QUEUE statement from a HTCONDOR submit file. There is an implicit TRANSFORM 1 at the end of a rules file that has no explicit TRANSFORM command.

OSGs default route expressed in the TJ's proposed new transform language.

lines starting with the keywords set, delete, copy, rename, name are commands to the transform engine and are executed as they are read. lines that are of the form key = value set temporary variables that can be referenced in transform commmands using the various $() and $function() macro expansions that are common to config and submit files. 

new transform language
NAME OSG CE Default route

MaxIdleJobs = 2000
MaxJobs = 10000

# by default, accept all jobs
Requirements = True

# these triggers control IF statements later in the transform
tmp.ExpireJob = False
tmp.RemoveIfIdle = False


# modify routed job attributes
#
DELETE CondorCE
SET    RoutedJob True

# remove routed job if the client disappears for 48 hours or it is idle for 6
#
IF $(tmp.RemoveIfIdle)
  SET    PeriodicRemove (LastClientContact - time() > 48*60*60) || (JobStatus == 1 && (time() - QDate) > 6*60)
ELSE
  DELETE PeriodicRemove
ENDIF

# insert HOME and OSG_* into environment
#
tmp.osg_env = OSG_GRID='/etc/osg/wn-client/' OSG_SQUID_LOCATION='fermicloud133.fnal.gov:3128' OSG_SITE_READ='None' OSG_APP='/share/osg/app' OSG_GLEXEC_LOCATION='None' OSG_DATA='UNAVAILABLE'
tmp.osg_env = $(tmp.osg_env) OSG_HOSTNAME='fermicloud136.fnal.gov' OSG_STORAGE_ELEMENT='False' OSG_SITE_NAME='herp' GLOBUS_LOCATION='/usr' OSG_WN_TMP='None' OSG_DEFAULT_SE='None' OSG_SITE_WRITE='None'
SET osg_environment "$(tmp.osg_env)"

tmp.user_home_expr = userHome(Owner, "/")
tmp.user_home = HOME=$EVAL(tmp.user_home_expr)

COPY Environment orig_environment
SET  Environment "$(tmp.user_home) $(MY.orig_environment) $(MY.osg_environment)"

# pick up GlobusRSL settings, we will use those later in the transform
# NOTE: is it a bug to leave this attribute behind?
#
# set InputRSL = ifThenElse(GlobusRSL is null, [], eval_rsl(GlobusRSL));
# or possibly this
IF DEFINED MY.GlobusRSL
  SET InputRSL eval_rsl(GlobusRSL)
ELSE
  SET InputRSL []
ENDIF

# Set new requirements
IF $(tmp.ExpireJob)
  SET Requirements (LastClientContact - time()) < 30*60
ELSE
  SET Requirements True
ENDIF

# pass attributes (maxMemory,xcount,jobtype,queue)
# via gWMS Factory described within ClassAd if undefined via RSL
# Note default memory request of 2GB
#
IF DEFINED MY.InputRSL.MaxMemory
  SET RequestMemory $(MY.InputRSL.MaxMemory)
ELIF $(MY.MaxMemory)
  SET RequestMemory MaxMemory
ELSE
  SET RequestMemory $(MY.default_maxMemory:2000)
ENDIF

IF DEFINED MY.InputRSL.Queue
  SET remote_queue "$(MY.InputRSL.Queue)"
ELIF DEFINED MY.Queue
  SET remote_queue Queue
ELSE
  SET remote_queue "$(MY.default_queue)"
ENDIF

# Figure out the number of cores. HTCondor uses RequestCpus
# blahp uses SMPGranularity and NodeNumber.  Default is 1 core.
#
IF DEFINED MY.InputRSL.xcount
  tmp.cpus = $(MY.InputRSL.xcount)
ELIF $(MY.xcount)
  tmp.cpus = $(MY.xcount)
ELSE
  tmp.cpus = $(MY.default_xcount:1)
ENDIF
SET RequestCpus $(tmp.cpus)
SET SMPGranularity $(tmp.cpus)
SET NodeNumber $(tmp.cpus)

# If remote_cerequirements is a string, BLAH will parse it as an expression before examining it
#
SET remote_cerequirements "CONDOR_CE == 1"

# add a walltime to the remote_cerequirements expression if one is given.
tmp.MaxWalltime_expr = 60 * (InputRSL.MaxWalltime ?: MaxWalltime ?: default_MaxWalltime ?: 0)
IF $INT(tmp.MaxWalltime_expr)
  SET remote_cerequirements "Walltime == $INT(tmp.MaxWalltime_expr) && CondorCE == 1"
ENDIF

For reference here is the same route expressed in the current new classad syntax 

current job router language
[ MaxIdleJobs = 2000;
 MaxJobs = 10000;
 /* by default, accept all jobs */
 Requirements = True;

 /* now modify routed job attributes */

 /* remove routed job if the client disappears for 48 hours or it is idle for 6 */
 /*set_PeriodicRemove = (LastClientContact - time() > 48*60*60) || (JobStatus == 1 && (time() - QDate) > 6*60); */
 delete_PeriodicRemove = true;
 delete_CondorCE = true;
 set_RoutedJob = true;
 copy_environment = "orig_environment";
 set_osg_environment = "OSG_GRID='/etc/osg/wn-client/' OSG_SQUID_LOCATION='fermicloud133.fnal.gov:3128' OSG_SITE_READ='None' OSG_APP='/share/osg/app' OSG_GLEXEC_LOCATION='None' OSG_DATA='UNAVAILABLE' OSG_HOSTNAME='fermicloud136.fnal.gov' OSG_STORAGE_ELEMENT='False' OSG_SITE_NAME='herp' GLOBUS_LOCATION='/usr' OSG_WN_TMP='None' OSG_DEFAULT_SE='None' OSG_SITE_WRITE='None'";
 eval_set_environment = debug(strcat("HOME=",
                              userHome(Owner, "/"),
                              " ",
                              ifThenElse(orig_environment is undefined,
                                         osg_environment,
                                         strcat(osg_environment, " ", orig_environment) )));

 /* Set new requirements */
 /* set_requirements = LastClientContact - time() < 30*60;*/
 set_requirements = True;

 set_InputRSL = ifThenElse(GlobusRSL is null, [], eval_rsl(GlobusRSL));

 /* Note default memory request of 2GB */
 /* Note yet another nested condition allow pass attributes (maxMemory,xcount,jobtype,queue) via gWMS Factory described within ClassAd if undefined via RSL */
 eval_set_RequestMemory = ifThenElse(InputRSL.maxMemory isnt null,
                                     InputRSL.maxMemory,
                                     ifThenElse(maxMemory isnt null,
                                                maxMemory,
                                                ifThenElse(default_maxMemory isnt null,
                                                           default_maxMemory, 2000)));
 eval_set_remote_queue = ifThenElse(InputRSL.queue isnt null,
                                    InputRSL.queue,
                                    ifThenElse(queue isnt null,
                                               queue,
                                               ifThenElse(default_queue isnt null, default_queue, "")));

 /* HTCondor uses RequestCpus;
 blahp uses SMPGranularity and NodeNumber.  Default is 1 core. */
 eval_set_RequestCpus = ifThenElse(InputRSL.xcount isnt null,
                                   InputRSL.xcount,
                                   ifThenElse(xcount isnt null,
                                              xcount,
                                              ifThenElse(default_xcount isnt null, default_xcount, 1)));
 eval_set_remote_SMPGranularity = ifThenElse(InputRSL.xcount isnt null,
                                             InputRSL.xcount,
                                             ifThenElse(xcount isnt null,
                                                        xcount,
                                                        ifThenElse(default_xcount isnt null, default_xcount, 1)));
 eval_set_remote_NodeNumber = ifThenElse(InputRSL.xcount isnt null,
                                         InputRSL.xcount,
                                         ifThenElse(xcount isnt null,
                                                    xcount,
                                                    ifThenElse(default_xcount isnt null, default_xcount, 1)));

 /* If remote_cerequirements is a string, BLAH will parse it as an expression before examining it */
 eval_set_remote_cerequirements = ifThenElse(InputRSL.maxWalTlime isnt null,
                                             strcat("Walltime == ",string(60*InputRSL.maxWallTime)," && CondorCE == 1"),
                                             ifThenElse(maxWallTime isnt null,
                                                        strcat("Walltime == ",string(60*maxWallTime)," && CondorCE == 1"),
                                                        ifThenElse(default_maxWallTime isnt null,
                                                                   strcat("Walltime == ", string(60*default_maxWallTime), " && CondorCE == 1"),
                                                                   "CondorCE == 1")));
]