HTCondorWiki: Chtc Scale Test Setup

 Our goal was to establish an HTCondor pool with 10,000 EC2 instances.  To conserve cash, but still test the pool at the expected scale, we requested Spot instance types in order of increasing cost (that is, starting with a single core), but configured that startd to always advertise 10 static slots.  Thus, with 10,000 instances, we would have a 100,000-slot pool.
 
-The following configuration, excepting the explanatory comments, is identical to the one successfully started 100,000 12-hour sleep jobs.
+The following configuration, excepting the explanatory comments, is identical to the one successfully started > 100,000 12-hour sleep jobs.
+
+We used shared port on the submitter and CCB on the execute nodes to minimize port usage on the submitter.  (We used shared port on the execute nodes to simplify security group configuration.)
 
 {section: general configuration}
 
@@ -103,7 +105,7 @@
 EC2_GAHP_DEBUG = D_PERF_TRACE D_SUB_SECOND
 {endverbatim}
 
-We repeated the following section 256 times.
+I repeated the following section 256 times.
 
 =--- $(LOCAL_CONFIG_DIR)/99-collector-tree ---=
 {verbatim}
@@ -167,3 +169,31 @@
 # Useless, but shuts up the master.
 COLLECTOR10000_LOG = $(LOG)/10000Log
 {endverbatim}
+
+{section: submitter}
+
+We dedicated a machine to being the submit node.  To save memory, we used the 32-bit shadow.  (We had to install an additional 32-bit compatibility library in order to run it without a wrapper script.)  Nonetheless, HTCondor required ~97 of 128 GB of RAM to run all 101,000 jobs.
+
+=--- $(LOCAL_CONFIG_DIR)/01-submitter ---=
+{verbatim}
+# We're a submitter.
+DAEMON_LIST = MASTER, SCHEDD
+CONDOR_HOST = $(FULL_HOSTNAME)
+
+# Shared-nothing.
+UID_DOMAIN = $(FULL_HOSTNAME)
+FILESYSTEM_DOMAIN = $(FULL_HOSTNAME)
+
+# Security.
+SEC_CLIENT_AUTHENTICATION_METHODS = FS, PASSWORD
+SEC_DEFAULT_AUTHENTICATION = REQUIRED
+SEC_DEFAULT_AUTHENTICATION_METHODS = FS, PASSWORD
+SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = TRUE
+SEC_PASSWORD_FILE = $(LOCAL_DIR)/password_file
+ALLOW_WRITE = condor_pool@*/* $(FULL_HOSTNAME) $(IP_ADDRESS) 127.0.0.1
+
+# Primary collector.
+COLLECTOR_HOST = e141.chtc.wisc.edu:9999
+{endverbatim}
+
+{section: execute node}