{section: Introduction}

The =condor_annex= tool rents computational resources from Amazon's cloud service and joins them to an HTCondor pool for your jobs to use.  These instructions document how to use =condor_annex= for CHTC jobs.  Some restrictions apply:

*: At the moment, you can only use =condor_annex= for jobs on =submit-4.chtc.wisc.edu=.
*: Your jobs must have =MayUseAWS= set in their ads.
*: Your jobs must have =WantFlocking= set in their ads.
*: Your jobs' requirements must, of course, allow them to run on Amazon's cloud.

These restrictions will be covered in the following instructions.

{section: Overview}

1: Prepare your AWS account
1:: Obtaining an Access Key
1:: Running the Set-Up Command
1:: Checking the Set-Up
2: Submit a Test Job
3: Run =condor_annex=
4: Running Jobs at Amazon
5: Cleaning Up (optional)

These instructions assume this is the first time you're using =condor_annex= on CHTC.

{section: 1 Prepare your AWS account}

The =condor_annex= tool includes a =-setup= command which will prepare your AWS account.  If you're not sure if you've done the set-up before, it won't hurt to repeat it, but you may save some time if you follow section 1.3 ("Checking the Set-Up") first.  If the set-up checks out OK, great; if not, return here and start at section 1.1 ("Obtaining an Access Key").

{subsection: 1.1 Obtaining an Access Key}

In order to use AWS, =condor_annex= needs a pair of security tokens (like a user name and password).  Like a user name, the "access key" is (more or less) public information; the corresponding "secret key" is like a password and must be kept a secret.  To help keep both halves secret, =condor_annex= (and HTCondor) are never told these keys directly; instead, you tell HTCondor which file to look in to find each one.

Log into =submit-4.chtc.wisc.edu= and create those two files now; we'll tell you how to fill them in shortly.  By convention, these files exist in your =~/.condor= directory, which is where =condor_annex -setup= will store the rest of the data it needs.

{term}
$ mkdir ~/.condor
$ cd ~/.condor
$ touch publicKeyFile privateKeyFile
$ chmod 600 publicKeyFile privateKeyFile
{endterm}

The last command ensures that only you can read or write to those files.

To donwload a new pair of security tokens for =condor_annex= to use, go to the {link: https://console.aws.amazon.com/iam/home?region=us-east-1#/users IAM console}; log in if you need to.  The following instructions assume you are logged in as a user with the privilege to create new users.  (The 'root' user for any account has this privilege; other accounts may as well.)

1: Click the "Add User" button.
1: Enter name in the *User name* box; "annex-user" is a fine choice.
1: Click the check box labelled "Programmatic access".
1: Click the button labelled "Next: Permissions".
1: Select "Attach existing policies directly".
1: Type "AdministratorAccess" in the box labelled "Filter".
1: Click the check box on the single line that will appear below (labelled "AdministratorAccess").
1: Click the "Next: review" button (you may need to scroll down).
1: Click the "Create user" button.
1: From the line labelled "annex-user", copy the value in the column labelled "Access key ID" to =publicKeyFile=.
1: On the line labelled "annex-user", click the "Show" link in the column labelled "Secret access key"; copy the revealed value to =privateKeyFile=.
1: Hit the "Close" button.

The 'annex-user' now has full privileges to your account.  We're working on creating a CloudFormation template that will create a user with only the privileges =condor_annex= actually needs.

{subsection: 1.2 Running the Set-Up Command}

The following command will set-up your AWS account.  It will create a number of persistent components, none of which will cost you anything to keep around.  These components can take quite some time to create; =condor_annex= checks each for completion every ten seconds and prints an additional dot (past the first three) when it does so, to let you know that everything's still working.

{term}
$ condor_annex -setup
Creating configuration bucket (this takes less than a minute)....... complete.
Creating Lambda functions (this takes about a minute)........ complete.
Creating instance profile (this takes about two minutes)................... complete.
Creating security group (this takes less than a minute)..... complete.
Setup successful.
{endterm}

{subsection: 1.3 Checking the Setup}

You can verify at this point (or any later time) that the set-up procedure completed successfully by running the following command.

{term}
$ condor_annex -check-setup
Checking for configuration bucket... OK.
Checking for Lambda functions... OK.
Checking for instance profile... OK.
Checking for security group... OK.
{endterm}

{section: 2 Submit a Test Job}

You haven't requested any resources yet, but ...

{section: 3 Run condor_annex}
{section: 4 Altering your Existing Jobs}

{section: 5 Cleaning Up (Optional)}

The resources =condor_annex= rents for you from Amazon will, as we mentioned before, shut themselves down after the duration, or if they're idle for longer than the time-out.  At that point, no more charges will accrue -- it costs you nothing to leave your account set-up to use =condor_annex=.

If, however, you want to be tidy, you may delete the components setup created by going to the {link: https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks?filter=active CloudFormation console} and deleting the entries whose names begin with 'HTCondorAnnex-'.  The setup procedure also creates an SSH key pair which may be useful for debugging; the private key was stored in =~/.condor/HTCondorAnnex-KeyPair.pem=.  To remove the corresponding public key from your AWS account, go to the {link: https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#KeyPairs:sort=keyName key pair console} and delete the 'HTCondorAnnex-KeyPair' key.