HTCondorWiki: Gridmanager Log

 {section: Reading the Gridmanager Log}
 
-{subsection: Normal Logging Level}
+{section: Normal Logging Level}
 
 This section describes entries that always appear in the gridmanager log.
 
@@ -53,7 +53,7 @@
 11/08/11 23:42:44 [15864] **** condor_gridmanager (condor_GRIDMANAGER) pid 15864 EXITING WITH STATUS 0
 {endcode}
 
-{subsection: Verbose Logging Level}
+{section: Verbose Logging Level}
 
 When something goes wrong, it's useful to increase the verbosity of the gridmanager's logging to collect more information. When you do so, you may also want to increase the maximum log size
 {code}
@@ -63,6 +63,8 @@
 
 This section will describe some of the additional information that will appear in the log when D_FULLDEBUG is enabled.
 
+{subsection: GAHP}
+
 All data exchanged with GAHP servers is logged. The gridmanager communicates with a GAHP server over a pair of pipes. The GAHP acknowledges the issuance of each command. The results of the command are supplied later, in response to a RESULTS command.
 
 {code}
@@ -83,6 +85,8 @@
 09/21/11 15:26:28 [4205] GAHP[4209] (stderr) -> 2011-09-21 15:26:28,909 INFO - CreamProxy_Delegate::execute() - Signing returned proxy with our proxy certificate [/tmp/x509up_u17063]...
 {endcode}
 
+{subsection: Status and attribute changes}
+
 The gridmanager logs when it observes the job's status as reported by the remote system change:
 {code}
 11/09/11 00:03:25 [16311] (717.0) globus state change: ACTIVE -> STAGE_OUT
@@ -101,3 +105,47 @@
 11/03/11 16:44:47 [29399]    NumGlobusSubmits = 1
 11/03/11 16:44:47 [29399] leaving doContactSchedd()
 {endcode}
+
+{subsection: State machine}
+
+The gridmanager maintains a state machine for each job. Transitions between states are usually triggered by events related to the job. Most states involve some action being taken on the job, with a transition triggered by completion of the action.
+
+Any time something happens that may affect a job, it's state machine is evaluated by calling the function doEvaluateState().
+{code}
+11/03/11 16:46:56 [29399] (714.933) doEvaluateState called: gmState GM_DONE_COMMIT, globusState 8
+{endcode}
+
+This may result in one or more transitions in the state machine.
+{code}
+11/03/11 16:46:56 [29399] (714.933) gm state change: GM_DONE_COMMIT -> GM_DELETE
+{endcode}
+
+The state machine is different for each scheduler type. However, there is a lot of similarity. Here are the states a GRAM or CREAM job will transition through during a normal job execution.
+
+*: GM_INIT: Finish initialization of job object
+*: GM_START: Decide whether we have a new job or are recovering a previous job submission
+*: GM_CLEAR_REQUEST: Clear job ad of any attributes relating to a previous job submission, wait for schedd to write changes to disk
+*: GM_UNSUBMITTED: Job ad is clean and ready for a new submission attempt
+*: GM_DELEGATE_PROXY: (CREAM only) Delegate proxy to CREAM, if not done already
+*: GM_SET_LEASE: (CREAM only) Set job lease with CREAM, if not done already
+*: GM_SUBMIT: Start job submission (CE can't start running job yet)
+*: GM_SUBMIT_SAVE: Save GridJobId persistently in schedd
+*: GM_STAGE_IN: (CREAM only) Transfer input files
+*: GM_SUBMIT_COMMIT: Finish job submission (CE is free to start running job)
+*: GM_SUBMITTED: Wait for CE to finish running job
+*: GM_PROBE_JOBMANAGER/GM_POLL_JOB_STATE: Ask CE for current status of job, return to GM_SUBMITTED afterwards
+*: GM_PUT_TO_SLEEP: (GRAM2 only) Kill jobmanager process
+*: GM_JOBMANAGER_ASLEEP: (GRAM2 only) Jobmanager is dead, listening to grid monitor
+*: GM_RESTART: (GRAM2 only) Restart jobmanager, usually because job is done
+*: GM_RESTART_SAVE: (GRAM2 only) Save new jobmanager URL in schedd
+*: GM_RESTART_COMMIT: (GRAM2 only) Tell restarted jobmanager to start managing job
+*: GM_CHECK_OUTPUT: (GRAM only) Verify output is completely transferred
+*: GM_STAGE_OUT: (CREAM only) Transfer output files
+*: GM_DONE_SAVE: Record job as completed in schedd
+*: GM_DONE_COMMIT: Tell CE to clean up job state
+*: GM_DELETE: Delete job object in gridmanager (terminal state)
+
+If an error occurs, the job may end up in one of these states:
+
+*: GM_CANCEL: Remove job from CE
+*: GM_HOLD: Place job on hold for user or admin to debug