{subsection: How Frequently to Checkpoint}
-Obviously, the longer the code spends writing checkpoints, and the longer your job spends transferring them, the longer it will take for you to get the job's results. Conversely, the more frequently the job transfers new checkpoints, the less time the job loses if it's interrupted. For most users and for most jobs, taking a checkpoint about once an hour works well, and it's not a bad place to start experimenting. A number of factors will skew this interval up or down:
+Obviously, the longer the code spends writing checkpoints, and the longer your job spends transferring them, the longer it will take for you to get the job's results. Conversely, the more frequently the job transfers new checkpoints, the less time the job loses if it's interrupted. For most users and for most jobs, taking a checkpoint about once an hour works well, and it's not a bad duration to start experimenting with. A number of factors will skew this interval up or down:
*: If your job(s) usually run on resources with strict time limits, you may want to adjust how often your job checkpoints to minimize wasted time. For instance, if your job writes a checkpoint after each hour, and each checkpoint takes five minutes to write out and then transfer, your fifth checkpoint will finish twenty-five minutes into the fifth hour, and you won't gain any benefit from the next thirty-five minutes of computation. If you instead write a checkpoint every eighty-four minutes, your job will only waste four minutes.
*: If a particular code writes larger checkpoints, or writes smaller checkpoints unusually slowly, you may want to take a checkpoint less frequently than you would for other jobs of a similar length, to keep the total overhead (delay) the same. The opposite is also true: if the job can take checkpoints particularly quickly, or the checkpoints are particularly small, the job could checkpoint more often for the same amount of overhead.
-*: Some code naturally checkpoints at longer or shorter intervals. If a code writes a checkpoint every five minutes, it may make sense for the =executable= to wait for the code to write ten or more checkpoints before exiting (which asks HTCondor to transfer the checkpoint file(s)). If a job is a sequence of steps, the natural (or only) checkpoint interval may be between steps.
+*: Some code naturally checkpoints at longer or shorter intervals. If a code writes a checkpoint every five minutes, it may make sense for the =executable= to wait for the code to write ten or more checkpoints before exiting (which asks HTCondor to transfer the checkpoint file(s)). If a job is a sequence of steps, the natural (or only possible) checkpoint interval may be between steps.
*: How long it takes to restart from a checkpoint. It should never take longer to restart from a checkpoint than to recompute from the beginning, but the restart process is part of the overhead of taking a checkpoint. The longer a code takes to restart, the less often the =executable= should exit.
Measuring how long it takes to make checkpoints is left as an exercise for the reader. Since version 8.9.1, however, HTCondor will report in the job's log (if a log is enabled for that job) how long file transfers, including checkpoint transfers, took.
