We recommend that you install a personal condor to make use of =condor_annex=; it's simpler to configure that way. Get started by following the instructions for CreatingPersonalHtcondor; be sure to download one of the tarballs for version 8.7.0 or later.
-For the rest of these instructions, where you see =LOCAL_DIR=, replace it with the LOCAL_DIR defined by the installation instructions above; the value used in the examples was =/scratch/local/condor84=.
+For the rest of these instructions, where you see =LOCAL_DIR=, replace it with the LOCAL_DIR defined by the installation instructions above; the value used in the examples was =/scratch/local/condor84=. Likewise, where you see =RELEASE_DIR=, replace it with the RELEASE_DIR defined by the installation instructions above; the value used in the examples was =/scratch/condor84=.
{subsection: Configure a pool password}
@@ -32,7 +32,7 @@
ALLOW_DAEMON = condor_pool@*
{endfile}
-You must also run the following command, which prompts you to enter a password:
+You also need to run the following command, which prompts you to enter a password:
{term}
$ condor_store_cred -c add
@@ -53,75 +53,132 @@
SHARED_PORT_PORT = 9618
{endfile}
-----
+{section: Prepare your AWS account}
-{section: Prepare your EC2 account}
+The current version of =condor_annex= still needs a little help from you to do its job. There are five pieces that need to be placed in the cloud; we provide three of the pieces, but you need to put them in place for us. (AWS will create the other two for you when you ask.) Instructions for each of these pieces follows; don't worry if you don't know what any of them mean or do -- the instructions will explain what you need to know.
-You will need to provide HTCondor with an access key/secret key pair. For security reasons, you specify the location of a file containing the secret key instead of specifying the secret key directly; the same goes for the access key. If you don't already have these keys, you can create new pair from the AWS web console; the process varies depending on which kind of account you have. (FIXME: (link to) Instructions for the root account.)
+1: A (private) S3 bucket
+1: An EC2 instance profile
+1: An AWS Lambda function
+1: A security group
+1: An SSH key pair.
-{subsection: Create a private S3 bucket}
+We'll be using the "us-east-1" region throughout.
-You'll need access to a private S3 bucket. (FIXME: (link to) Instructions for creating private bucket.) In the following instructions, we'll call this bucket 'privateBucketName'; replace that string, when you see it, with the actual name of the private bucket you created in this step.
+{subsection: Create a (private) S3 bucket}
-{subsection: Prepare the lease machineery}
+An S3 bucket is a place in the AWS cloud where you can store files. =condor_annex= stores the dynamic configuration the instances in your annex will need in an S3 bucket. If the bucket is private, than only you can read the files in it -- allowing your instances, and only your instances, to read those files is what the next step is for. These two steps make it possible for =condor_annex= to securely share the password you entered earlier.
-To avoid having to upload it every time, the annex assumes that the Lambda function its needs already exists and is configured to run as a role with the required permissons. We've provided a CloudFormation template that will create and configure the Lambda function for you [FIXME: where?]. Instructions follow for readers who haven't created a stack from a template file before. After logging into the AWS web console, do the following for each region you intend to use (you do just 'us-east-1' to start, since that has the example AMI):
+To create an S3 bucket, go to the {link: https://console.aws.amazon.com/s3/home?region=us-east-1 S3 console}; log in if you need to. Then:
+
+1: Click the "Create Bucket" button.
+1: Enter a name at the prompt. Amazon makes this harder than it needs to be by requiring that the bucket name is unique. A name like 'annex-<username>-<year>-<month>-<day>' (where you replace <things> with their actual values) has a good chance of being unique; =condor_annex= does not require a particular style of name. Where you see =s3PrivateBucket= in the following instructions, replace it with the name you entered here.
+1: Select the "US Standard" region.
+1: Click the "Create" button.
+
+Thankfully, the default for newly-created bucket is to be private.
+
+{subsection: Create an EC2 instance profile}
+
+On EC2, an "instance profile" is a way to associate a "role" with an instance. A "role" is collection of privileges that the instance would not otherwise have. Specifically, each annex instance needs to have the privilege to download its dynamic configuration from the otherwise-private S3 bucket you just created. We've created a CloudFormation template which creates the instance profile and role for you; it can be found in =RELEASE_DIR/FIXME=. If you're not running your web browser on your Linux machine, you'll need to copy this file from the latter to the former.
+
+To create the instance profile, go to the {link: https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks?filter=active CloudFormation} console; log in if you need to. Then:
-1: Switch to the region. (The second drop-down box in from the upper right.)
-1: Switch to CloudFormation. (In the _Services_ menu, under _Management_.)
1: Click the "Create Stack" button.
1: Upload the template using the "Browse..." button
1: Click the "Next" button.
-1: Name the stack; "HTCondorLeaseImplementation" is a good name.
+1: Name the stack; "HTCondorAnnexInstanceProfile" is a good name.
1: Click the "Next" button. (You don't to change anything on the options screen.)
1: Check the box next to "I acknowledge" (down near the bottom) and click the "Create" button (where the "Next" button was).
-1: AWS should return the list of stacks; select the one you just created and select the "outputs" tab.
-1: Copy the long string labelled "LeaseFunctionARN"; you'll need it to configure _condor_annex_. It may take some time for that string to appear (you may need to reload the page, as well.) Wait the stack to enter the 'CREATE_COMPLETE' state before using the LeaseFunctionARN (see below).
-
-{subsection: Prepare the dynamic configuration machinery}
-
-For the same reason, you'll have to create a role for the annex instances, so they (but nobody else) can access the private S3 bucket. [FIXME: This should probably just be CF parameter?] Use the =generate-role= script, distributed FIXME, to create a CloudFormation template:
+1: AWS will display a list of stacks; wait for the one you just created (it may be the only one) to enter the "CREATE_COMPLETE" state. You may need to refresh your browser.
+1: Select the stack you just created (click on the status rather than the name), and then select the "Outputs" tab.
+1: Copy the long string labelled "InstanceConfigurationProfile"; where you see =InstanceConfigurationProfile= in the following instructions, replace it with the string you copied here.
-{term}
-$ generate-role privateBucketName config.tar.gz > role.json
-{endterm}
+{subsection: Create an AWS Lambda function}
-Create a stack by uploading =role.json=, but otherwise following the instructions from the previous section; the stack's output will be named "InstanceConfigurationProfile", and you'll need its value later.
+An AWS Lambda function is a way of running (usually small snippets of) code on AWS without starting an instance. =condor_annex= uses this ability to ensure that the duration you specify when starting an annex is not exceeded, even if the Linux machine is longer running when the lease expires. We've created a CloudFormation template which creates and configures the Lambda function for you; it can be found in your RELEASE_DIR/FIXME. If you're not running your web browser on your Linux machine, you'll need to copy this file from the latter to the former.
-{subsection: Create Spot Fleet role}
+To create the instance profile, go to the {link: https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks?filter=active CloudFormation} console; log in if you need to. Then (these instructions should look familiar):
-If this account you're using has never created a Spot Fleet, create one now:
-
-1: Switch to the region. (The second drop-down box in from the upper right.)
-1: Switch to EC2. (In the _Services_ menu, under _Compute_.)
-1: Click on "Spot Requests" in the list on the left (under _INSTANCES_).
-1: Click the "Request Spot Instances" button.
+1: Click the "Create Stack" button.
+1: Upload the template using the "Browse..." button
1: Click the "Next" button.
-1: [FIXME: automagic creating the IAM Fleet Role].
+1: Name the stack; "HTCondorAnnexLambdaFunction" is a good name.
+1: Click the "Next" button. (You don't to change anything on the options screen.)
+1: Check the box next to "I acknowledge" (down near the bottom) and click the "Create" button (where the "Next" button was).
+1: AWS will display a list of stacks; wait for the one you just created to enter the "CREATE_COMPLETE" state. You may need to refresh your browser.
+1: Select the stack you just created (click on the status rather than the name), and then select the "Outputs" tab.
+1: Copy the long string labelled "LeaseFunctionARN"; where you see =LeaseFunctionARN= in the following instructions, replace it with the string you copied here.
+
+{subsection: Create a security group}
-{subsection: Create Security Group}
+On AWS, a "security group" is a set of firewall rules; it defines which machines can use the services your instance may provide. Specifically, for the annex to work, you'll need to allow (at least) the Linux machine you're using to connect to the instances' HTCondor service. (These instructions include an open port for SSH, as well, just in case that becomes necessary.) For simplicity, these instructions allow connections to the SSH and HTCondor services from anywhere. If you know the public IP address of the Linux machine you're using, you can enter that address instead of selecting "Anywhere" in those two steps (select "Custom" and type the address in the box).
-You'll also need a security group that allows HTCondor (and SSH, just in case) through the firewall:
+To create the security group, go to the {link: https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#SecurityGroups:sort=groupId security group} console; log in if you need to. Then:
-1: Click on "Security Groups" (under _NETWORK & SECURITY_).
1: Click the "Create Security Group" button.
1: Enter a name; "HTCondorAnnexSG" is a good one.
1: Enter a description; "Allows SSH and HTCondor from anywhere" would be accurate.
1: Make sure that the "Inbound" tab (the default) is selected.
-1: Click the "Add rule" button. Change the left-most drop-down box from "Custom TCP Rule" to "SSH"; change the right-most drop-down box from "Custom" to "Anywhere". (This is less secure than it could be, but these instructions don't have room for a discussion about that.)
+1: Click the "Add rule" button. Change the left-most drop-down box from "Custom TCP Rule" to "SSH"; change the right-most drop-down box from "Custom" to "Anywhere".
1: Click the "Add rule" button again. This time, change the second text box to read "9618" (its column is labelled "Port Range"); also change the right-most drop-down box from "Custom" to "Anywhere".
1: Click the "Create" button.
-1: You'll now see a list of security groups. The second column is the group name; find the name you entered when you created the group ("HTCondorAnnexSG") and record its security group ID (the column to the left).
+
+You'll now see a list of security groups. The second column is the group name; find the name you entered when you created the group ("HTCondorAnnexSG") and record its security group ID (the column to the left). Where you see =SecurityGroupID= in the following instructions, replace it with the ID you recorded here.
+
+{subsection: Create an SSH key pair}
+
+You'll only need this if something goes wrong, but in that case, you'll need it really badly. An SSH key pair allows you to log in (via SSH) to an instance started with the specified key pair. (You probably used SSH to log in to the Linux machine you've been using.)
+
+Go to the {link: https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#KeyPairs:sort=keyName EC2 console}; log in if you need to. Then:
+
+1: Click the "Create Key Pair" button.
+1: Name the key pair "annex-key-pair".
+1: Click the "Create" button.
+1: Your browser will automatically prompt you about downloading a file called "annex-key-pair.pem". Save this file somewhere private.
+
+If you transfer that file to the Linux machine, you can run =ssh -i annex-key-pair.pem ec2-user@instanceAddr= to log in; non-Linux SSH clients should be able to use the .pem file (and log in as 'ec2-user') as well, but those instructions are out of scope for this document. (You can obtain an instance's address from the {link: https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Instances:sort=instanceState EC2 console}.)
{section: Configure condor_annex}
-These instructions use an image published by the HTCondor developers to help people get started. The image's OS is Amazon Linux (based on CentOS 6). The example =config.json= [FIXME: Where?] uses that image and generates slots with 1 CPU and 2 GB of RAM using whatever instance type(s) happen to be cheapest at the time. If you want to tweak those values, read the AWS docs: http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_SpotFleetRequestConfigData.html. The example =config.json= bids the on-demand prices for its instance types. See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html for more information about Spot prices and bidding.
+Almost there! You need to get one more pair of things from AWS before you can fully configure =condor_annex=.
+
+{subsection: Obtaining an Access Key}
+
+In order to use AWS, =condor_annex= needs a pair of security tokens (like a user name and password). Like a user name, the "access key" is (more or less) public information; the corresponding "secret key" is like a password and must be kept a secret. To help keep both halves secret, =condor_annex= (and HTCondor) are never told these keys directly; instead, you tell HTCondor which file to look in to find each one.
+
+Create those two files now; we'll tell you how to fill them in shortly. The following is a reasonable way to do so. If you'd prefer they'd be in a particular directory, feel free to change the first line.
+
+{term}
+$ cd
+$ touch accessKeyFile
+$ touch secretKeyFile
+$ chmod 600 accessKeyFile secretKeyFile
+$ pwd
+{endterm}
+
+The second-to-last command ensures that only you can read or write to those files. When you see =path/to/accessKeyFile= or =path/to/secretKeyFile= in the following instructions, replace =path/to= with the line printed by the last command.
+
+To donwload a new pair of security tokens for =condor_annex= to use, go to the {link: https://console.aws.amazon.com/iam/home?region=us-east-1#/users IAM console}; log in if you need to. The following instructions assume you are logged in as a user with the privilege to create new users. (The 'root' user for any account has this privilege; other accounts may as well.)
+
+1: Click the "Add User" button.
+1: Enter name in the *User name* box; "annex-user" is a fine choice.
+1: Click the check box labelled "Programmatic access".
+1: Click the button labelled "Next: Permissions".
+1: Select "Attach existing policies directly".
+1: Type "AdministratorAccess" in the box labelled "Filter".
+1: Click the check box on the single line that will appear below (labelled "AdministratorAccess").
+1: Click the "Next: review" button (you may need to scroll down).
+1: Click the "Create user" button.
+1: From the line labelled "annex-user", copy the value in the column labelled "Access key ID" to =accessKeyFile=.
+1: On the line labelled "annex-user", click the "Show" link in the column labelled "Secret access key"; copy the revealed value to =secretKeyFile=.
+1: Hit the "Close" button.
-_[We think we can automate the process of configuring HTCondor and its image down to just installed a special "EC2" RPM but the last attempt had a bug.]_
+The 'annex-user' now has full privileges to your account. We're working on creating a CloudFormation template that will create a user with only the privileges =condor_annex= actually needs.
-Add the following lines to the HTCondor configuration:
+{subsection: putting it all together}
-----
+Add the following lines to the =LOCALDIR/condor_config.local=:
{file: LOCALDIR/condor_config.local}
# These following lines are common to all accounts, and are included
@@ -136,8 +193,8 @@
ANNEX_DEFAULT_S3_URL = https://s3.amazonaws.com
# All subsequent lines are specific to your particular account.
-ANNEX_DEFAULT_ACCESS_KEY_FILE =
-ANNEX_DEFAULT_SECRET_KEY_FILE =
+ANNEX_DEFAULT_ACCESS_KEY_FILE = path/to/accessKeyFile
+ANNEX_DEFAULT_SECRET_KEY_FILE = path/to/secretKeyFile
# The following lines configure the image and type of on-demand instance
# that condor_annex will use if you don't specify otherwise. Note that
@@ -147,9 +204,9 @@
ANNEX_DEFAULT_ODI_IMAGE_ID = ami-aacfc2bd
# The instance profile must "point" to the S3_CONFIG_PATH.
-ANNEX_DEFAULT_ODI_INSTANCE_PROFILE_ARN =
-ANNEX_DEFAULT_ODI_S3_CONFIG_PATH =
-ANNEX_DEFAULT_ODI_LEASE_FUNCTION_ARN =
-ANNEX_DEFAULT_ODI_KEY_NAME =
-ANNEX_DEFAULT_ODI_SECURITY_GROUP_IDS =
+ANNEX_DEFAULT_ODI_INSTANCE_PROFILE_ARN = instanceProfileArn
+ANNEX_DEFAULT_ODI_S3_CONFIG_PATH = privateBucketName/config.tar.gz
+ANNEX_DEFAULT_ODI_LEASE_FUNCTION_ARN = leaseFunctionArn
+ANNEX_DEFAULT_ODI_KEY_NAME = annex-key-pair
+ANNEX_DEFAULT_ODI_SECURITY_GROUP_IDS = securityGroupID
{endfile}
