{subsection: Expectations}
-1. Your job can [be configured to] exit after taking a checkpoint with an exit code it does not otherwise use.
-2. When restarted, your job determines on its own if a checkpoint is available, and if so, uses it.
-3. Starting your job up from a checkpoint is relatively quick.
-4. Your job can not otherwise communicate with HTCondor, or it does not atomically update its checkpoint(s).
+1: Your job can [be configured to] exit after taking a checkpoint with an exit code it does not otherwise use.
+2: When restarted, your job determines on its own if a checkpoint is available, and if so, uses it.
+3: Starting your job up from a checkpoint is relatively quick.
+4: Your job can not otherwise communicate with HTCondor, or it does not atomically update its checkpoint(s).
{subsection: Using +WantFTOnCheckpoint}
@@ -27,13 +27,10 @@
{subsection: Meeting the Expectations}
-1. If your job can be made to exit after taking a checkpoint, but does not return a unique exit code when doing so, a wrapper script for the job may be able to inspect the sandbox after the job exits and determine if a checkpoint was successfully created. If so, this wrapper script could then return a unique value. If the job can return literally any value at all, HTCondor can also regard being killed by a particular (unique) signal as a sign of a successful checkpoint; set =+SuccessCheckpointExitBySignal= to =TRUE= and =+SuccessCheckpointExitSignal= to the particular signal. If your job can not be made to exit after taking a checkpoint, a wrapper script may be able to determine when a successful checkpoint has been taken and kill the job itself.
-
-2. If your job requires different arguments to start from a checkpoint, you can wrap your job in a script which checks for the presence of a checkpoint and runs the jobs with the corresponding arguments.
-
-3. If it takes too long to start your job up from a checkpoint, you will need to take checkpoints less often to make quicker progress. This, of course, increases the risk of losing a substantial amount of work when your job is interrupted.
-
-4. If a job does not update its checkpoints atomically, you can use a wrapper script to atomically update some other file(s) and treat those as the checkpoints. More specifically, if your job writes a checkpoint incrementally to a file called 'checkpoint', but exits with code 17 when it's finished doing so, your wrapper script can check for exit code 17 and then rename 'checkpoint' to 'checkpoint.atomic'. Because rename is an atomic operation, this will prevent HTCondor from transferring a partial checkpoint file, even if the job is interrupted in the middle of taking a checkpoint. Your wrapper script will also have to copy 'checkpoint.atomic' to 'checkpoint' before starting the job, so that the job (a) uses the safe checkpoint file and (b) doesn't corrupt that checkpoint file if interrupted at a bad time.
+1: If your job can be made to exit after taking a checkpoint, but does not return a unique exit code when doing so, a wrapper script for the job may be able to inspect the sandbox after the job exits and determine if a checkpoint was successfully created. If so, this wrapper script could then return a unique value. If the job can return literally any value at all, HTCondor can also regard being killed by a particular (unique) signal as a sign of a successful checkpoint; set =+SuccessCheckpointExitBySignal= to =TRUE= and =+SuccessCheckpointExitSignal= to the particular signal. If your job can not be made to exit after taking a checkpoint, a wrapper script may be able to determine when a successful checkpoint has been taken and kill the job itself.
+2: If your job requires different arguments to start from a checkpoint, you can wrap your job in a script which checks for the presence of a checkpoint and runs the jobs with the corresponding arguments.
+3: If it takes too long to start your job up from a checkpoint, you will need to take checkpoints less often to make quicker progress. This, of course, increases the risk of losing a substantial amount of work when your job is interrupted.
+4: If a job does not update its checkpoints atomically, you can use a wrapper script to atomically update some other file(s) and treat those as the checkpoints. More specifically, if your job writes a checkpoint incrementally to a file called 'checkpoint', but exits with code 17 when it's finished doing so, your wrapper script can check for exit code 17 and then rename 'checkpoint' to 'checkpoint.atomic'. Because rename is an atomic operation, this will prevent HTCondor from transferring a partial checkpoint file, even if the job is interrupted in the middle of taking a checkpoint. Your wrapper script will also have to copy 'checkpoint.atomic' to 'checkpoint' before starting the job, so that the job (a) uses the safe checkpoint file and (b) doesn't corrupt that checkpoint file if interrupted at a bad time.
Future versions of HTCondor may provide for explicit coordination between the job and HTCondor. Modifying a job to explicitly coordinate with HTCondor would substantially alter the expectations.
