Page History

Turn Off History

One approach to using cloud resources to run HTCondor jobs is to use the condor_annex tool to expand an existing pool onto the cloud (see HowToUseCondorAnnexWithOnDemandInstances). Another approach, documented here, is to create a new HTCondor pool entirely in the cloud. The HTCondor team maintains an AWS Marketplace entry to help simplify the process[, but target-audience disclaimer]. These instructions assume you already have an AWS account and a key pair (specifically, the private half).

Overview

The general approach will be to use the Marketplace entry to start a head node, which will be the brains of the new HTCondor pool, as well as be where you'll log in to and submit jobs from. Once the head node is up and running, you'll use condor_annex to add cloud resources to your new pool. Then you can start running jobs, and when they're done, shut everything down.

[FIXME: subsections, linkage]

  1. Starting a Head Node
  2. Adding Cloud Resources to Your New Pool
  3. Running Jobs
  4. Cleaning Up

Starting a Head Node

  1. Open HTCondor's Marketplace entry in another tab.
  2. Click the orange 'Continue' button to the right. You may need to log in to AWS at this point.
  3. This is a busy page, but there's only one thing you may have to change on it: the "key pair" setting, which is all the way down at the bottom. Change the selected key pair, if necessary, to be one whose private half you have.
  4. This step is where you will start spending money. Scroll back up; there will be section to the right titled "price for your selections." That's what Amazon will charge starting when you click the orange 'Launch with one-click' button.

Adding Cloud Resource to Your New Pool

Running Jobs

Cleaning Up