Ticket #2428: allow macros in dagman log files to assist templating

Mentioned on condor-users:

Universe   = standard
requirements = (Arch == "X86_64")&&  (OpSys == "LINUX")
Executable = /path/to/binary
transfer_executable = TRUE
should_transfer_files = IF_NEEDED
when_to_transfer_output = ON_EXIT
Arguments  = $(param)
Output     = $(id).stdout
Error      = $(id).stderr
Log        = $(id).condor_log
Queue

NODE_STATUS_FILE dag_status
JOBSTATE_LOG     dag_log
JOB   RT000-rt0.01 itersave.condor_job
VARS  RT000-rt0.01 param="000 GT"
VARS  RT000-rt0.01 id="RT000-rt0.01"
RETRY RT000-rt0.01 3
JOB   RT001-rt0.01 itersave.condor_job
VARS  RT001-rt0.01 param="001 GT"
VARS  RT001-rt0.01 id="RT001-rt0.01"
RETRY RT001-rt0.01 3
(and more)

MultiLogFiles: macros ('$(...') not allowed in log file name
($(id).condor_log) in DAG node submit files

DAGMan strictly protects against possibly changing UserLog files because of the presence of macros. This covers $(Cluster) and $(Process). However, it also covers against VARS ID=ABC, as in this example, a macro that is not variable and instead allows for template submit files.

Proposal is to more narrowly disallow macros (nix Process/Cluster/anything-that-evals-at-submit-time-with-Process-or-Cluster-or-Time-etc) or give users who know what they are doing an escape hatch, a way to say "yes, i'm using macros properly, trust me"

[Append remarks]

Remarks:

2011-Nov-17 12:56:19 by wenger:
I'm marking this as stalled because if we actually implement #2645, this ticket will become irrelevant.


2011-Nov-21 16:20:02 by wenger:
Deferring this because of the ideas in #2645 -- if we implement some of those things, DAGMan may no longer have to parse the log file names.


2012-Oct-04 14:23:07 by wenger:
This is pretty much no longer an issue because of #2807 being done. I'm going to leave it unresolved for now, but move it down to priority 5. I can probably be abandoned once the new "one log file" feature is widely used.
[Append remarks]

Properties:

Type: enhance           Last Change: 2012-Oct-18 14:29
Status: resolved          Created: 2011-Aug-29 09:11
Fixed Version: v070900           Broken Version: v070000 
Priority:          Subsystem: Dag 
Assigned To: nwp           Derived From:  
Creator: matt  Rust:  
Customer Group: users  Visibility: public 
Notify: wenger@cs.wisc.edu, matt@cs.wisc.edu,tstclair@redhat.com  Due Date:  

Derived Tickets:

#2772   Macro in node job log file name should be fatal error

Related Check-ins:

2012-Jul-12 12:04   Check-in [32747]: edits of version history items and defn of knob DAGMAN_ALWAYS_USE_NODE_LOG ===GT=== #2428 ===GT=== #2807 (By Karen Miller )
2012-May-02 22:18   Check-in [31904]: Let submit use recursive macros ===GT=== #2807 This has been suggested a lot. This is a partial solution to #406 and #2428 (By Nathan W. Panike )
2012-May-02 22:18   Check-in [31907]: Use new macro capability in DAGman DAGman now requests the schedd and shadow to write each event to the default log, i.e., <DAG file>.nodes.log, along with whatever log is specified by the user. [...] (By Nathan W. Panike )
2011-Sep-29 16:54   Check-in [28312]: Working on gittrac #2428: cleaned up the VARS code in preparation for allowing macros in the log file specifications; also started on the first test for this. (By Kent Wenger )