{section: condor_transform_ads language}

Transform rules files consist of lines containing key=value pairs or
transform commands such as =SET=, =RENAME=, etc. Transform commands execute
as they are read and can make use of values set up until that point
using the =$(key)= macro substitution commands that HTCondor configuration files
and condor_submit files use.

Most constructs that work in these files will also work in rules files such as if/else. Macro substitution will fetch attributes of the ClassAd to be transformed when =$(MY.attr)= is used.

The transform commands are:

|=SET     <attr> <expr>=|Set =<attr>= to =<expr>=|
|=EVALSET <attr> <expr>=|Evaluate =<expr>= and then set =<attr>= to the result|
|=DEFAULT <attr> <expr>=|Set =<attr>= to =<expr>= if =<attr>= is undefined or missing|
|=COPY    <attr> <newattr>=|Copy the value of =<attr>= to =<newattr>=|
|=COPY    /<regex>/ <newattrs>=|Copy the values of attributes whose names  match =<regex>= to =<newattrs>=|
|=RENAME  <attr> <newattr>=|Rename =<attr>= to =<newattr>=|
|=RENAME  /<regex>/ <newattrs>=|Rename attributes matching =<regex>= to =<newattrs>=|
|=DELETE  <attr> <newattr>=|Delete =<attr>=|
|=DELETE  /<regex>/=|Delete attributes matching =<regex>=|
|=EVALMACRO <key> <expr>=|Evaluate =<expr>= and then insert it as a transform macro value|

Optionally, a transform rule set can end with an iteration command

=TRANSFORM [<N>] [<vars>] [in <list> {quote:|} from <file> {quote:|} matching <pattern>]=

In the above commands =<attr>= must be a valid attribute name and =<expr>= a valid {quote: ClassAd}
expression or literal.  The various $() macros will be expanded in =<expr>=, =<newattr>= or
=<newattrs>= before they are parsed as {quote: ClassAd} expressions or attribute names.

When a =COPY=,=RENAME= or =DELETE= with =<regex>= is used, regex capture groups are substituted in
=<newattrs>= after $() expansion. \0 will expand to the entire match, \1 to the first capture, etc.

A =TRANSFORM= command must be the last command in the rules file. It takes the same options as the
=QUEUE= statement from a HTCONDOR submit file. There is an implicit =TRANSFORM 1= at the end of a rules file
that has no explicit =TRANSFORM= command.


{subsection: OSGs default route expressed in the TJ's proposed new transform language. }

lines starting with the keywords *set*, *delete*, *copy*, *rename*, *name* are commands to the transform engine and are executed as they are read. lines that are of the form key = value set temporary variables that can be referenced in transform commmands using the various $() and $function() macro expansions that are common to config and submit files.

{snip: new transform language}
NAME OSG CE Default route

MaxIdleJobs = 2000
MaxJobs = 10000

# by default, accept all jobs
Requirements = True

# these triggers control IF statements later in the transform
xf.ExpireJob = false
xf.RemoveIfIdle = false


# modify routed job attributes
#
delete CondorCE
set    RoutedJob true

# remove routed job if the client disappears for 48 hours or it is idle for 6
#
if $(xf.RemoveIfIdle)
  set    PeriodicRemove (LastClientContact - time() > 48*60*60) || (JobStatus == 1 && (time() - QDate) > 6*60)
else
  delete PeriodicRemove
endif

# insert HOME and OSG_* into environment
#
xf.osg_env = OSG_GRID='/etc/osg/wn-client/' OSG_SQUID_LOCATION='fermicloud133.fnal.gov:3128' OSG_SITE_READ='None' OSG_APP='/share/osg/app' OSG_GLEXEC_LOCATION='None' OSG_DATA='UNAVAILABLE'
xf.osg_env = $(xf.osg_env) OSG_HOSTNAME='fermicloud136.fnal.gov' OSG_STORAGE_ELEMENT='False' OSG_SITE_NAME='herp' GLOBUS_LOCATION='/usr' OSG_WN_TMP='None' OSG_DEFAULT_SE='None' OSG_SITE_WRITE='None'
set osg_environment = "$(xf.osg_env)"

xf.user_home_expr = userHome(Owner, "/")
xf.user_home = HOME=$EVAL(xf.user_home_expr)

copy Environment orig_environment
set Environment = "$(xf.user_home) $(my.orig_environment) $(my.osg_environment)"

# pick up GlobusRSL settings, we will use those later in the transform
# NOTE: is it a bug to leave this attribute behind?
#
# set InputRSL = ifThenElse(GlobusRSL is null, [], eval_rsl(GlobusRSL));
# or possibly this
if defined MY.GlobusRSL
  set InputRSL = eval_rsl(GlobusRSL)
else
  set InputRSL = []
endif

# Set new requirements
if $(xf.ExpireJob)
  set Requirements (LastClientContact - time()) < 30*60
else
  set Requirements True
endif

# pass attributes (maxMemory,xcount,jobtype,queue)
# via gWMS Factory described within ClassAd if undefined via RSL
# Note default memory request of 2GB
#
if defined MY.InputRSL.MaxMemory
  Set RequestMemory = $(MY.InputRSL.MaxMemory)
elif $(MY.MaxMemory)
  Set RequestMemory = MaxMemory
else
  Set RequestMemory = $(MY.default_maxMemory:2000)
endif

if defined MY.InputRSL.Queue
  Set remote_queue = "$(MY.InputRSL.Queue)"
elif defined MY.Queue
  Set remote_queue = Queue
else
  Set remote_queue = "$(MY.default_queue)"
endif

# Figure out the number of cores. HTCondor uses RequestCpus
# blahp uses SMPGranularity and NodeNumber.  Default is 1 core.
#
if defined MY.InputRSL.xcount
  xf.cpus = $(MY.InputRSL.xcount)
elif $(MY.xcount)
  xf.cpus = $(MY.xcount)
else
  xf.cpus = $(MY.default_xcount:1)
endif
Set RequestCpus = $(xf.cpus)
Set SMPGranularity = $(xf.cpus)
Set NodeNumber = $(xf.cpus)

# If remote_cerequirements is a string, BLAH will parse it as an expression before examining it
#
set remote_cerequirements = "CONDOR_CE == 1"

# add a walltime to the remote_cerequirements expression if one is given.
xf.MaxWalltime_expr = 60 * (InputRSL.MaxWalltime ?: MaxWalltime ?: default_MaxWalltime ?: 0)
if $INT(xf.MaxWalltime_expr)
  set remote_cerequirements = "Walltime == $INT(xf.MaxWalltime_expr) && CondorCE == 1"
endif
{endsnip}


For reference here is the same route expressed in the current new classad syntax
{snip: current job router language}
[ MaxIdleJobs = 2000;
 MaxJobs = 10000;
 /* by default, accept all jobs */
 Requirements = True;

 /* now modify routed job attributes */

 /* remove routed job if the client disappears for 48 hours or it is idle for 6 */
 /*set_PeriodicRemove = (LastClientContact - time() > 48*60*60) || (JobStatus == 1 && (time() - QDate) > 6*60); */
 delete_PeriodicRemove = true;
 delete_CondorCE = true;
 set_RoutedJob = true;
 copy_environment = "orig_environment";
 set_osg_environment = "OSG_GRID='/etc/osg/wn-client/' OSG_SQUID_LOCATION='fermicloud133.fnal.gov:3128' OSG_SITE_READ='None' OSG_APP='/share/osg/app' OSG_GLEXEC_LOCATION='None' OSG_DATA='UNAVAILABLE' OSG_HOSTNAME='fermicloud136.fnal.gov' OSG_STORAGE_ELEMENT='False' OSG_SITE_NAME='herp' GLOBUS_LOCATION='/usr' OSG_WN_TMP='None' OSG_DEFAULT_SE='None' OSG_SITE_WRITE='None'";
 eval_set_environment = debug(strcat("HOME=",
                              userHome(Owner, "/"),
                              " ",
                              ifThenElse(orig_environment is undefined,
                                         osg_environment,
                                         strcat(osg_environment, " ", orig_environment) )));

 /* Set new requirements */
 /* set_requirements = LastClientContact - time() < 30*60;*/
 set_requirements = True;

 set_InputRSL = ifThenElse(GlobusRSL is null, [], eval_rsl(GlobusRSL));

 /* Note default memory request of 2GB */
 /* Note yet another nested condition allow pass attributes (maxMemory,xcount,jobtype,queue) via gWMS Factory described within ClassAd if undefined via RSL */
 eval_set_RequestMemory = ifThenElse(InputRSL.maxMemory isnt null,
                                     InputRSL.maxMemory,
                                     ifThenElse(maxMemory isnt null,
                                                maxMemory,
                                                ifThenElse(default_maxMemory isnt null,
                                                           default_maxMemory, 2000)));
 eval_set_remote_queue = ifThenElse(InputRSL.queue isnt null,
                                    InputRSL.queue,
                                    ifThenElse(queue isnt null,
                                               queue,
                                               ifThenElse(default_queue isnt null, default_queue, "")));

 /* HTCondor uses RequestCpus;
 blahp uses SMPGranularity and NodeNumber.  Default is 1 core. */
 eval_set_RequestCpus = ifThenElse(InputRSL.xcount isnt null,
                                   InputRSL.xcount,
                                   ifThenElse(xcount isnt null,
                                              xcount,
                                              ifThenElse(default_xcount isnt null, default_xcount, 1)));
 eval_set_remote_SMPGranularity = ifThenElse(InputRSL.xcount isnt null,
                                             InputRSL.xcount,
                                             ifThenElse(xcount isnt null,
                                                        xcount,
                                                        ifThenElse(default_xcount isnt null, default_xcount, 1)));
 eval_set_remote_NodeNumber = ifThenElse(InputRSL.xcount isnt null,
                                         InputRSL.xcount,
                                         ifThenElse(xcount isnt null,
                                                    xcount,
                                                    ifThenElse(default_xcount isnt null, default_xcount, 1)));

 /* If remote_cerequirements is a string, BLAH will parse it as an expression before examining it */
 eval_set_remote_cerequirements = ifThenElse(InputRSL.maxWalTlime isnt null,
                                             strcat("Walltime == ",string(60*InputRSL.maxWallTime)," && CondorCE == 1"),
                                             ifThenElse(maxWallTime isnt null,
                                                        strcat("Walltime == ",string(60*maxWallTime)," && CondorCE == 1"),
                                                        ifThenElse(default_maxWallTime isnt null,
                                                                   strcat("Walltime == ", string(60*default_maxWallTime), " && CondorCE == 1"),
                                                                   "CondorCE == 1")));
]
{endsnip}