How to upgrade HTCondor gracefully

Known to work with HTCondor version: 7.0

There are 2 approaches to upgrading to a newer version of HTCondor:

  1. One way to gracefully upgrade is to shut down the pool, install the new version of HTCondor, and then start it back up. To do that, see How to shut down HTCondor without killing jobs. However, before you do that, consider the consequence of waiting for jobs to finish. On multi-core machines, if all cores but one are idle, because you are waiting for a job to finish, this may be worse than killing everything and quickly restarting.

  2. Another way to upgrade leaves HTCondor running. HTCondor will automatically restart itself if the condor_master binary is updated. To take advantage of this, configure HTCondor so that the path to binaries (for example, MASTER) points to the new binaries. One way to do that (under Unix) is to use a symbolic link that points to the current HTCondor installation directory (e.g. /opt/condor). Once the new files are in place, change the symbolic link to point to the new directory. If HTCondor is configured to locate its binaries via the symbolic link, then after the symbolic link changes, the condor_master daemon will notice the new binaries and restart itself. How frequently it checks is controlled by the configuration variable MASTER_CHECK_NEW_EXEC_INTERVAL, which defaults 5 minutes.

How to check what version of HTCondor is running on all machines in a pool

On Unix platforms, the following is a handy way to summarize the HTCondor versions that exist in the pool:

condor_status -master -format "%s\n" CondorVersion | sort | uniq -c