Recovery mode is a mode in which DAGMan "catches up" with what has happened with the node jobs of its DAG before actually submitting any new jobs (it does the "catching up" by reading the user log files of the node jobs). Basically, we get into recovery mode if a DAG was running and DAGMan exited in a way that didn't create a rescue DAG, and then DAGMan was restarted. There are two main ways that this can happen: -*: condor_hold/condor_release of the condor_dagman job. +*: condor_hold/condor_release of the condor_dagman job (condor_hold actually kills the condor_dagman process, but leaves the job in the queue; condor_release starts a new DAGMan process, which goes into recovery mode). *: DAGMan exits without creating a rescue DAG because of something like an assertion. You can also force DAGMan into recovery mode by creating an appropriate lock file and condor_submitting the .condor.sub file (instead of condor_submit_dagging the DAG file).
