When the master notices new binaries, it begins a graceful restart, which may not be exactly what you want. On an execute machine, a graceful restart means that running jobs are preempted. Standard universe jobs will attempt to checkpoint, which could be a problem if all machines in a large pool attempt to do this at the same time. If they do not complete within the cutoff time specified by the KILL policy expression (default 10 minutes), then they are killed without checkpointing. You may therefore want to increase this cutoff time and you may also want to upgrade the pool in stages rather than all at once.
-For universes other than standard universe, jobs are preempted. If jobs have been guaranteed a certain amount of uninterrupted run time with MaxJobRetirementTime, then the job is not killed until the specified amount of retirement time has been exceeded (it's 0 by default). The first step of killing the job is a soft kill signal, which can be intercepted by the job so that it can shut down gracefully, save state etc. If the job has not gone away once the KILL expression fires (10 minutes by default), then it is forcibly hard-killed. Since graceful shutdown of jobs may rely on shared resources such as disks where state is saved, the same reasoning applies as for standard universe: you may want to increase the KILL time for large pools and you may want to upgrade the pool in stages to avoid jobs running out of time.
+For universes other than standard universe, jobs are preempted. If jobs have been guaranteed a certain amount of uninterrupted run time with {quote: MaxJobRetirementTime}, then the job is not killed until the specified amount of retirement time has been exceeded (it's 0 by default). The first step of killing the job is a soft kill signal, which can be intercepted by the job so that it can shut down gracefully, save state etc. If the job has not gone away once the KILL expression fires (10 minutes by default), then it is forcibly hard-killed. Since graceful shutdown of jobs may rely on shared resources such as disks where state is saved, the same reasoning applies as for standard universe: you may want to increase the KILL time for large pools and you may want to upgrade the pool in stages to avoid jobs running out of time.
Another time limit to be aware of is the configuration setting {code}SHUTDOWN_GRACEFUL_TIMEOUT{endcode}. This defaults to 30 minutes. If the graceful restart is not completed within this time, a fast restart ensues. This causes jobs to be hard-killed.
