Using Condor Annex For The First Time Eight Seven One

Not logged in

• Browse
• Help
• Home
• Login
• Reports
• Search
• Timeline
• Wiki

• Contents
• No-Diff
• Text

 
 {section: Prepare your AWS account}
 
-The current version of =condor_annex= still needs a little help from you to do its job.  There are five pieces that need to be placed in the cloud; we provide three of the pieces, but you need to put them in place for us.  (AWS will create the other two for you when you ask.)  Instructions for each of these pieces follows; don't worry if you don't know what any of them mean or do -- the instructions will explain what you need to know.
-
-1: A (private) S3 bucket
-1: An EC2 instance profile
-1: A pair of AWS Lambda functions
-1: A security group
-1: An SSH key pair.
-
-We'll be using the "us-east-1" region throughout.
-
-The last step in using =condor_annex= for the first time is to tell =condor_annex=, via HTCondor configuration, about the things you did in these five steps (and a sixth one, below).  You may find it easiest to copy and paste the example configuration from that last step ("putting it all together" -- look there for where to paste) and update it as you go.
-
-{subsection: Create a (private) S3 bucket}
-
-An S3 bucket is a place in the AWS cloud where you can store files.  =condor_annex= stores the dynamic configuration the instances in your annex will need in an S3 bucket.  If the bucket is private, than only you can read the files in it -- allowing your instances, and only your instances, to read those files is what the next step is for.  These two steps make it possible for =condor_annex= to securely share the password you entered earlier.
-
-To create an S3 bucket, go to the {link: https://console.aws.amazon.com/s3/home?region=us-east-1 S3 console}; log in if you need to.  Then:
-
-1: Click the "Create Bucket" button.
-1: Enter a name at the prompt.  Amazon makes this harder than it needs to be by requiring that the bucket name is unique.  A name like 'annex-<username>-<year>-<month>-<day>' (where you replace <things> with their actual values) has a good chance of being unique; =condor_annex= does not require a particular style of name.  Where you see =s3PrivateBucket= in the following instructions, replace it with the name you entered here.
-1: Select the "US Standard" region.
-1: Click the "Create" button.
-
-Thankfully, the default for newly-created bucket is to be private.
-
-{subsection: Create an EC2 instance profile}
-
-On EC2, an "instance profile" is a way to associate a "role" with an instance.  A "role" is collection of privileges that the instance would not otherwise have.  Specifically, each annex instance needs to have the privilege to download its dynamic configuration from the otherwise-private S3 bucket you just created.  We've created a CloudFormation template which creates the instance profile and role for you.
-
-To create the instance profile, go to the {link: https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks?filter=active CloudFormation} console; log in if you need to.  Then:
-
-1: Click the "Create Stack" button.
-1: Choose "Specify an Amazon S3 template URL" and enter "https://s3.amazonaws.com/condor-annex/role-7.json".
-1: Click the "Next" button.
-1: Name the stack; "HTCondorAnnexInstanceProfile" is a good name.
-1: Type =s3PrivateBucket= in the field labeled "S3BucketName".
-1: Click the "Next" button.
-1: You don't need to change anything on this (the "Options") page.  Click the "Next" button.
-1: Check the box next to "I acknowledge" (down near the bottom) and click the "Create" button (where the "Next" button was).
-1: AWS will display a list of stacks; wait for the one you just created (it may be the only one) to have "CREATE_COMPLETE" in the column labelled "Status".  You may need to refresh (use the circling-arrow button) to update the list.
-1: Select the stack you just created (click on the status rather than the name), and then select the "Outputs" tab.
-1: Copy the long string labelled "InstanceProfileARN"; where you see =InstanceProfileARN= in the following instructions, replace it with the string you copied here.
-
-{subsection: Create a pair of AWS Lambda functions}
-
-An AWS Lambda function is a way of running (usually small snippets of) code on AWS without starting an instance.  =condor_annex= uses this ability to ensure that the duration you specify when starting an annex is not exceeded, even if the Linux machine is longer running when the lease expires.  We've created a CloudFormation template which creates and configures the Lambda function for you.
-
-To create the instance profile, go to the {link: https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks?filter=active CloudFormation} console; log in if you need to.  Then (these instructions should look familiar):
-
-1: Click the "Create Stack" button.
-1: Choose "Specify an Amazon S3 template URL" and enter "https://s3.amazonaws.com/condor-annex/template-7.json".
-1: Click the "Next" button.
-1: Name the stack; "HTCondorAnnexLambdaFunctions" is a good name.
-1: Type =s3PrivateBucket= in the field labeled "S3BucketName".
-1: Click the "Next" button.
-1: You don't need to change anything on this (the "Options") page.  Click the "Next" button.
-1: Check the box next to "I acknowledge" (down near the bottom) and click the "Create" button (where the "Next" button was).
-1: AWS will display a list of stacks; wait for the one you just created to enter the "CREATE_COMPLETE" state.  You may need to refresh (use the circling-arrow button) to update the list.
-1: Select the stack you just created (click on the status rather than the name), and then select the "Outputs" tab.
-1: Copy the long string labelled "odiLeaseFunctionARN"; where you see =odiLeaseFunctionARN= in the following instructions, replace it with the string you copied here.
-1: Copy the long string labelled "sfrLeaseFunctionARN"; where you see =sfrLeaseFunctionARN= in the following instructions, replace it with the string you copied here.
-
-{subsection: Create a security group}
-
-On AWS, a "security group" is a set of firewall rules; it defines which machines can use the services your instance may provide.  Specifically, for the annex to work, you'll need to allow (at least) the Linux machine you're using to connect to the instances' HTCondor service.  (These instructions include an open port for SSH, as well, just in case that becomes necessary.)  For simplicity, these instructions allow connections to the SSH and HTCondor services from anywhere.  If you know the public IP address of the Linux machine you're using, you can enter that address instead of selecting "Anywhere" in those two steps (select "Custom" and type the address in the box).
-
-To create the security group, go to the {link: https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#SecurityGroups:sort=groupId security group} console; log in if you need to.  Then:
-
-1: Click the "Create Security Group" button.
-1: Enter a name; "HTCondorAnnexSG" is a good one.
-1: Enter a description; "Allows SSH and HTCondor from anywhere" would be accurate.
-1: Make sure that the "Inbound" tab (the default) is selected.
-1: Click the "Add rule" button.  Change the left-most drop-down box from "Custom TCP Rule" to "SSH"; change the right-most drop-down box from "Custom" to "Anywhere".
-1: Click the "Add rule" button again.  This time, change the second text box to read "9618" (its column is labelled "Port Range"); also change the right-most drop-down box from "Custom" to "Anywhere".
-1: Click the "Create" button.
-
-You'll now see a list of security groups.  The second column is the group name; find the name you entered when you created the group ("HTCondorAnnexSG") and record its security group ID (the column to the left).  Where you see =SecurityGroupID= in the following instructions, replace it with the ID you recorded here.
-
-{subsection: Create an SSH key pair}
-
-You'll only need this if something goes wrong, but in that case, you'll need it really badly.  An SSH key pair allows you to log in (via SSH) to an instance started with the specified key pair.  (You probably used SSH to log in to the Linux machine you've been using.)
-
-Go to the {link: https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#KeyPairs:sort=keyName EC2 console}; log in if you need to.  Then:
-
-1: Click the "Create Key Pair" button.
-1: Name the key pair "annex-key-pair".
-1: Click the "Create" button.
-1: Your browser will automatically prompt you about downloading a file called "annex-key-pair.pem".  Save this file somewhere private.
-
-If you transfer that file to the Linux machine, you can run =ssh -i annex-key-pair.pem ec2-user@instanceAddr= to log in; non-Linux SSH clients should be able to use the .pem file (and log in as 'ec2-user') as well, but those instructions are out of scope for this document.  (You can obtain an instance's address from the {link: https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances:sort=instanceState EC2 console}.)
-
-{section: Configure condor_annex}
-
-Almost there!  You need to get one more pair of things from AWS before you can fully configure =condor_annex=.
+The =condor_annex= tool now includes a =-setup= command which will prepare your AWS account.
 
 {subsection: Obtaining an Access Key}
 
 In order to use AWS, =condor_annex= needs a pair of security tokens (like a user name and password).  Like a user name, the "access key" is (more or less) public information; the corresponding "secret key" is like a password and must be kept a secret.  To help keep both halves secret, =condor_annex= (and HTCondor) are never told these keys directly; instead, you tell HTCondor which file to look in to find each one.
 
-Create those two files now; we'll tell you how to fill them in shortly.  The following is a reasonable way to do so.  If you'd prefer they'd be in a particular directory, feel free to change the first line.
+Create those two files now; we'll tell you how to fill them in shortly.  By convention, these files exist in your =~/.condor= directory, which is where =condor_annex -setup= will store the rest of the data it needs.
 
 {term}
-$ cd
+$ mkdir ~/.condor
+$ cd ~/.condor
 $ touch accessKeyFile
 $ touch secretKeyFile
 $ chmod 600 accessKeyFile secretKeyFile
-$ pwd
 {endterm}
 
-The second-to-last command ensures that only you can read or write to those files.  When you see =path/to/accessKeyFile= or =path/to/secretKeyFile= in the following instructions, replace =path/to= with the line printed by the last command.
+The last command ensures that only you can read or write to those files.
 
 To donwload a new pair of security tokens for =condor_annex= to use, go to the {link: https://console.aws.amazon.com/iam/home?region=us-east-1#/users IAM console}; log in if you need to.  The following instructions assume you are logged in as a user with the privilege to create new users.  (The 'root' user for any account has this privilege; other accounts may as well.)
 
@@ -279,37 +186,12 @@
 
 The 'annex-user' now has full privileges to your account.  We're working on creating a CloudFormation template that will create a user with only the privileges =condor_annex= actually needs.
 
-{subsection: putting it all together}
+{subsection: Running the Setup Command}
 
-Add the following lines to the =LOCALDIR/condor_config.local=:
+The following command will setup your AWS account.  It will create a number of persistent components; if you decide not to use =condor_annex=, you may delete them by going to the {link: https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks?filter=active CloudFormation console} and deleting the entries whose names begin with 'HTCondorAnnex-'.  It also creates an SSH key pair which may be useful for debugging; the private key is stored in =~/.condor/HTCondorAnnex-KeyPair.pem=.  To remove the corresponding public key from your AWS account, go to the {link: https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#KeyPairs:sort=keyName key pair console} and delete the 'HTCondorAnnex-KeyPair' key.
 
-{file: LOCALDIR/condor_config.local}
-# These following lines are common to all accounts, and are included
-# here in case you want to try using a different region.  You'll also
-# have to change the default AMI, or specify a different one when
-# invoking condor_annex.
-ANNEX_DEFAULT_EC2_URL = https://ec2.us-east-1.amazonaws.com
-ANNEX_DEFAULT_CWE_URL = https://events.us-east-1.amazonaws.com
-ANNEX_DEFAULT_LAMBDA_URL = https://lambda.us-east-1.amazonaws.com
-ANNEX_DEFAULT_S3_URL = https://s3.amazonaws.com
-
-# The following lines configure the image and type of on-demand instance
-# that condor_annex will use if you don't specify otherwise.  Note that
-# the default image ID set here is for us-east-1, so if you've changed
-# the region-specific URLs above, you'll need to change the ID here, too.
-ANNEX_DEFAULT_ODI_INSTANCE_TYPE = m4.large
-ANNEX_DEFAULT_ODI_IMAGE_ID = ami-83269195
-
-# All subsequent lines are specific to your particular account.
-ANNEX_DEFAULT_S3_BUCKET = s3PrivateBucket
-ANNEX_DEFAULT_ODI_INSTANCE_PROFILE_ARN = instanceProfileARN
-ANNEX_DEFAULT_SFR_LEASE_FUNCTION_ARN = sfrLeaseFunctionArn
-ANNEX_DEFAULT_ODI_LEASE_FUNCTION_ARN = odiLeaseFunctionArn
-ANNEX_DEFAULT_ODI_SECURITY_GROUP_IDS = securityGroupID
-ANNEX_DEFAULT_ODI_KEY_NAME = annex-key-pair
-
-ANNEX_DEFAULT_ACCESS_KEY_FILE = path/to/accessKeyFile
-ANNEX_DEFAULT_SECRET_KEY_FILE = path/to/secretKeyFile
-{endfile}
+{term}
+$ condor_annex -setup
+{endterm}
 
 You're ready to run =condor_annex=!  Return to HowToUseCondorAnnexWithOnDemandInstances.

This work is supported by NSF under Cooperative Agreement OAC-2030508 as part of the PATh Project. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF. Site built using CVSTrac.