Page History

Turn Off History

This page describes a configuration useful primarily for virtual machines which should be shut down when they stop being useful (doing work).

We start by configuring the startd to shut down if it hasn't been claimed for five minutes: 

STARTD_NOCLAIM_SHUTDOWN = 300.

We then configure the master to shut down when the startd shuts down. Note that we give the startd sixty seconds to start up the first time; this time-out would be unnecessary if STARTD_StartTime were guaranteed to be unset rather than zero before the startd starts up. 

MASTER.DAEMON_SHUTDOWN_FAST = ( STARTD_StartTime == 0 ) && ((CurrentTime - DaemonStartTime) > 60)

Then, when the master shuts down, we want to run a script to shut down the machine. All that this configuration knob does is define the "SHUTDOWN" name; you can pick another. 

MASTER_SHUTDOWN_SHUTDOWN = /etc/condor/shutdown

To activate the shutdown script named "SHUTDOWN", use the condor_set_shutdown command with the -exec option. This command must be run each time the master is (re)started by a user with sufficient privilege. Since start-up times are somewhat variable, you may want to use a retry loop like the one below; make sure that service and condor_set_shutdown are in the PATH. (If you've automated the start-up of HTCondor, you need to automate running condor_set_shutdown to make this work; if the HTCondor in your VM starts on boot, you could put this in rc.local, for instance.) 

while ! condor_set_shutdown -exec shutdown; do sleep 1; done

The script (/etc/condor/shutdown in this example) must of course be executable (and have the #! line), and is run by the master as root when the master shuts down. The following script may be useful as an example to work from. 

#!/bin/sh

# We start out trying to determine if this shutdown was voluntary or not.
# In this example, we don't do anything useful with this information, so
# you can skip it if you like.
#
# I'm not sure if /sbin/runlevel is available on systemd systems, so you
# check to make sure that this is normally nonzero and zero when shutting down.
SHUTDOWN=`/sbin/runlevel | /usr/bin/awk '{print $2}'`
SHUTDOWN_MESSAGE='because instance is being terminated'
if [ ${SHUTDOWN} -ne 0 ]; then
    SHUTDOWN_MESSAGE='for lack of work'
fi

# The following line is probably AWS-specific; you can omit it and the
# ${INSTANCE_ID} from the following line entirely.
INSTANCE_ID=$(/usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id)
MESSAGE="$(/bin/date) Instance ${INSTANCE_ID} shutting down ${SHUTDOWN_MESSAGE}."

# For testing.  You could do something cleverer here, if you'd like.
echo ${MESSAGE}

# Shut the machine down.  Comment this line out if you're just testing,
# of course. :)
/sbin/shutdown -h now