Page History
DAGMan monitors the state of submitted jobs solely by reading the user logs for the node jobs.
Things to mention:
- issues with macros in log file names
- multi-log code truncates files the first time it monitors them (except in recovery mode)
- multi-log code (really, lower-level code) keeps track of where we were in log files, in case we re-monitor them
- we monitor the log file for a node just before submitting it; unmonitor after job is done; special stuff for POST scripts (we write post script terminated event)
- multi-log code essentially coalesces any number of log files into a single stream of events
- submit events contain node name -- that's how we associate HTCondor IDs with nodes in recovery mode (in "normal" mode we get the HTCondor ID from the condor_submit output; I have to check what we do if that disagrees with the corresponding submit event when we read it)
- if we have two instances of the same DAG running at the same time (or any two DAG instances whose node jobs share user log files), things will get goofed up because the submit event notes only contain the node name; there's no way to tell which DAG the event goes with if they both have nodes with the given name
- possible confusion in recovery mode when going from DST to standard time, because timestamps are in local time
- ReadMultipleUserLogs class vs. MultiLogFiles class
- 1-sec sleeps before submits to make sure we can unambiguously order events when reading them back (because the user log timestamps only have a resolution of 1 sec)
- why are HTCondor and Stork events handled with separate ReadMultipleUserLogs objects? I don't remember -- need to look that up.
- default log used if no log specified by submit file -- log file passed on condor_submit_dag command line
- in the Dag class: ProcessLogEvents(), ProcessOneEvent(), ProcessAbortEvent(), ProcessTerminatedEvent(), etc.
- we don't care about all events
- Dag::EventSanityCheck()
- cases where events show up in incorrect order (e.g., execute before submit)