Creating Condor Pools At Amazon

Not logged in

• Browse
• Help
• Home
• Login
• Reports
• Search
• Timeline
• Wiki

• Contents
• No-Diff
• Text

 
 The general approach will be to use the Marketplace entry to start a _head node_, which will be the brains of the new HTCondor pool, as well as be where you'll log in to and submit jobs from.  Once the head node is up and running, you'll use =condor_annex= to add cloud resources to your new pool.  Then you can start running jobs, and when they're done, shut everything down.
 
-[FIXME: subsections, linkage]
-
 1: Start a Head Node
 2: Add Cloud Resources to Your New Pool
 2:: Log into your Head Node
@@ -14,8 +12,10 @@
 2:: Add Cloud Resources
 3: Run Jobs
 4: Clean Up
+4:: The Cloud Resources
+4:: The Head Node
 
-{section: Start a Head Node}
+{section: 1 Start a Head Node}
 
 1: Open HTCondor's {link: https://aws.amazon.com/marketplace/pp/B073WHVRPR Marketplace entry} in another tab.
 1: Click the orange 'Continue' button to the right.  You may need to log in to AWS at this point.
@@ -23,17 +23,17 @@
 1: This step is where you will start spending money.  Scroll back up; there will be section to the right titled "Price for your Selections."  That's what Amazon will charge starting when you click the orange 'Launch with one-click' button.  We'll remind you in these instructions, but you'll have to stop or terminate the instance your about to start on your own.
 1: On the next page, click on the "Your Software" link in the green box.
 
-{section: Add Cloud Resources to Your New Pool}
+{section: 2 Add Cloud Resources to Your New Pool}
 
-Your head node (by default) starts with two CPUs and 8 GiB of RAM.  Immediately after you log in -- see the next section -- you can start submitting and running jobs (see the section "Running Jobs", below).  However, with only two CPUs, you'll only be able to run two jobs at a time.  To add cloud resources to your pool, you'll use =condor_annex=.  To do that, you'll need to obtain an access key for =condor_annex=, so it can act interact with AWS on your behalf.  You'll only need to do that once.  Likewise, =condor_annex= has to do some one-time set-up for each account.  Once that's done, you can run =condor_annex= to add cloud resources as often as you like.
+Your head node (by default) starts with two CPUs and 8 GiB of RAM.  Immediately after you log in -- see the next section -- you can start submitting and running jobs (see section 3).  However, with only two CPUs, you'll only be able to run two jobs at a time.  To add cloud resources to your pool, you'll use =condor_annex=.  To do that, you'll need to obtain an access key for =condor_annex=, so it can act interact with AWS on your behalf.  You'll only need to do that once.  Likewise, =condor_annex= has to do some one-time set-up for each account.  Once that's done, you can run =condor_annex= to add cloud resources as often as you like.
 
-{subsection: Log into your Head Node}
+{subsection: 2.1 Log into your Head Node}
 
 1: Find the HTCondor entry and click the 'View Instances' button.
 1: There should only be one instance; click on the "Manage in AWS Console" link.  This will bring up the EC2 console with your head node selected.
 1: Right-click on the selected instance and select 'Connect'.  Follow the instructions, except replace =root@= with =ec2-user@=.  [FIXME]
 
-{subsection: Obtain an Access Key}
+{subsection: 2.2 Obtain an Access Key}
 
 Just being able to log into an EC2 instance doesn't give you the privilege to start additional EC2 instances.  In order to use add cloud resources to your new pool, HTCondor needs a pair of security tokens (like a user name and password). Like a user name, the "access key" is (more or less) public information; the corresponding "secret key" is like a password and must be kept a secret. To help keep both halves secret, you never tell HTCondor these keys directly; instead, you tell HTCondor which file to look in to find each one.
 
@@ -47,13 +47,13 @@
 
 The last command ensures that only you can read or write to those files.
 
-If you saved your security tokens from the last time you used =condor_annex=, copy them into the files you just created and skip to the section labelled "Running the Set-Up Command".
+If you saved your security tokens from the last time you used =condor_annex=, copy them into the files you just created and skip to section 2.3.
 
 To download a new pair of security tokens for =condor_annex= to use, go to the {link: https://console.aws.amazon.com/iam/home#users IAM console}; log in if you need to.
 
-{subsubsection: Privileged (Administrator) Accounts}
+{subsubsection: 2.2.1 Privileged (Administrator) Accounts}
 
-The following instructions assume you are logged in as a user with the privilege to create new users.  (The 'root' user for any account has this privilege; other accounts may as well.)  If your account has more limited privileges, skip past these instructions and follow the next set ("Non-Administrators").
+The following instructions assume you are logged in as a user with the privilege to create new users.  (The 'root' user for any account has this privilege; other accounts may as well.)  If your account has more limited privileges, skip to section 2.2.2.
 
 1: Click the "Add User" button.
 1: Enter name in the *User name* box; "annex-user" is a fine choice.
@@ -68,9 +68,9 @@
 1: On the line labelled "annex-user", click the "Show" link in the column labelled "Secret access key"; copy the revealed value to =privateKeyFile=.  Also copy this value your laptop or desktop computer; you'll want to have it if you use =condor_annex= again.
 1: Hit the "Close" button.
 
-The 'annex-user' now has full privileges to your account.  We're working on creating a CloudFormation template that will create a user with only the privileges =condor_annex= actually needs.
+The 'annex-user' now has full privileges to your account.  We're working on creating a CloudFormation template that will create a user with only the privileges =condor_annex= actually needs.  Skip to section 2.2.3.
 
-{subsubsection: Non-Privileged (Non-Administrator) Accounts}
+{subsubsection: 2.2.2 Non-Privileged (Non-Administrator) Accounts}
 
 If you're using an account with limited privileges, your administrator may have already given you the credentials.  If not, you may be able to create credentials for yourself at the {link: https://console.aws.amazon.com/iam/home#users IAM console}.
 
@@ -81,11 +81,11 @@
 1: Click the "Show" link in the column labelled "Secret access key"; copy the revealed value to =privateKeyFile=.  Also copy this value to your laptop or desktop computer; you'll want to have it if you use =condor_annex= again.
 1: Hit the "Close" button.
 
-{subsubsection: Save the Access Key}
+{subsubsection: 2.2.3 Save the Access Key}
 
 You should keep a copy both files somewhere safe, so you don't have to recreate them every time you start a new pool in the cloud.
 
-{subsection: Prepare your Account}
+{subsection: 2.3 Prepare your Account}
 
 The following command will prepare your AWS account for =condor_annex=.  It will create a number of persistent components, none of which will cost you anything to keep around.  These components can take quite some time to create; =condor_annex= checks each for completion every ten seconds and prints an additional dot (past the first three) when it does so, to let you know that everything's still working.
 
@@ -98,7 +98,7 @@
 Setup successful.
 {endterm}
 
-{subsubsection: Verify Account Preparation}
+{subsubsection: 2.3.1 Verify Account Preparation}
 
 You can verify at this point (or any later time) that the account-preparation procedure completed successfully by running the following command.
 
@@ -111,7 +111,7 @@
 Your setup looks OK.
 {endterm}
 
-{subsection: Add Cloud Resources}
+{subsection: 2.4 Add Cloud Resources}
 
 Run the following command; if you type 'yes', it will add ten instances to the pool for no more than 24 hours:
 
@@ -128,7 +128,7 @@
 
 {link: http://research.cs.wisc.edu/htcondor/manual/v8.7/6_Cloud_Computing.html Complete documentation} is also available.
 
-{section: Run Jobs}
+{section: 3 Run Jobs}
 
 To run on your new resources, a job's submit file must contain the following line:
 
@@ -142,9 +142,9 @@
 should_transfer_files = TRUE
 {endverbatim}
 
-{section: Clean Up}
+{section: 4 Clean Up}
 
-{subsection: The Cloud Resources}
+{subsection: 4.1 The Cloud Resources}
 
 One of the benefits of using =condor_annex= is that it will automatically terminate instances after a certain amount of time (24 hours in the example above).  This happens even if the instance is running a job at the time, to make sure that misbehaving jobs don't cause you to spend more than you intended.  Additionally, if at any time it's been too long (15 minutes by default) since an instance ran a job, it will shut itself down to save you money.  However, if you'd like to shut down the instances early, you can do so using the =condor_off= command, replacing =MyFirstAnnex= with the name of the annex you'd like to shut down:
 
@@ -152,6 +152,6 @@
 $ condor_off -annex MyFirstAnnex
 {endterm}
 
-{subsection: The Head Node}
+{subsection: 4.2 The Head Node}
 
 As noted above, you'll need to clean the head node up yourself.  If you don't want to keep any of your changes, then you should "terminate" the head node to avoid paying for storage.  If you just want to save money and pick up where you left off a bit later, you should instead "stop" the head node; you'll pay to keep its disk around until you start it again later.  Both options are under "Instance State" if you right-click on the instance in the EC2 console.

This work is supported by NSF under Cooperative Agreement OAC-2030508 as part of the PATh Project. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF. Site built using CVSTrac.