HTCondorWiki: How To Auto Retry Elsewhere

 Imagine that you knew your job will never run for longer than 20 minutes, and if your job does run for more than 20 minutes, you want to presume something is wrong with the machine. Here is an example of a condor_submit file that tells HTCondor to restart the job on a different machine if the job ran for more than 20 minutes.
 
-Note the technique is to put the job on hold via =periodic_hold= if it runs too long, resulting in the job going to the Held state.  Next the job is released via =periodic_release=, causing the job to go back to Idle and be rescheduled.  The =requirements= expression ensures the job runs on different machine entirely, not just a different slot on the same machine; see AvoidingBlackHoles.
-
-Also note that =periodic_release= expression only releases a job that was put on hold for a known cause, which we implement by utilizing the =periodic_hold_subcode= attribute.  After all, we don't want to release a job that was put on hold for a different reason, such as the user running =condor_hold=.  We also set =periodic_hold_reason= to something helpful, so typing =condor_q -hold= displays something informative.
-
-Finally, note that we limit the number of times a job goes through the hold/release cycle.
-
-All of the mechanisms used in the below submit file are described on {link: http://research.cs.wisc.edu/htcondor/manual/current/condor_submit.html the condor_submit manual page}.  Also may be useful to browse the {link: http://research.cs.wisc.edu/htcondor/manual/current/12_Appendix_A.html documented job classad attributes}.
-
 {code}
 # Fill in executable and max expected runtime in minutes.
 # If the job runs longer than expected, it will go on hold,
@@ -33,3 +25,11 @@
 periodic_hold_reason = ifthenelse(JobRunCount<3,"Ran too long, will retry","Ran too long")
 queue
 {endcode}
+
+Note the technique is to put the job on hold via =periodic_hold= if it runs too long, resulting in the job going to the Held state.  Next the job is released via =periodic_release=, causing the job to go back to Idle and be rescheduled.  The =requirements= expression ensures the job runs on different machine entirely, not just a different slot on the same machine; see AvoidingBlackHoles.
+
+Also note that =periodic_release= expression only releases a job that was put on hold for a known cause, which we implement by utilizing the =periodic_hold_subcode= attribute.  After all, we don't want to release a job that was put on hold for a different reason, such as the user running =condor_hold=.  We also set =periodic_hold_reason= to something helpful, so typing =condor_q -hold= displays something informative.
+
+Finally, note that we limit the number of times a job goes through the hold/release cycle.
+
+All of the mechanisms used in the below submit file are described on {link: http://research.cs.wisc.edu/htcondor/manual/current/condor_submit.html the condor_submit manual page}.  Also may be useful to browse the {link: http://research.cs.wisc.edu/htcondor/manual/current/12_Appendix_A.html documented job classad attributes}.