Jeff's Stork Responsibilities: Brain dump meeting Monday June 5, 3:30pm notes: *query Gnats category stork *fix security bugs first !!!!!!! query for category==Stork and sub-category==condor-security *Jaime's nanoHUB bug reports, feature requests complete, and merge V6_7-storkd-rewrite-branch (module=V6_7_CODE). This branch is a complete rewrite of stork server daemon. The goal of this effort was to take the original research-quality codebase up to a modern, production quality daemon. As of 2006-06-09, this rewrite is incomplete, and approx 60% done. general improvements in V6_7-storkd-rewrite-branch: *remove hundreds of fixed length buffers *improved fault protection *consolidation of many redundant code fragments into new Stork class methods. *consolidated multiple URL parsers into single new URL parser class. new features in V6_7-storkd-rewrite-branch: *added requirements and rank to job scheduling. Old stork scheduling was previously a simple FIFO. *checks all job attributes in submit file at submit time, not execution time *per-job run time sandbox directories *added throttle for starting new jobs *increased used of [new] ClassAd views [for better or worse ;-) ] to factor out many redundant static queries. *consolidated all stork job ad attributes into single file stork_job_ad.h *consolidated all stork config params into single file stork_config.h Gnats bug fixes: 570 hung job detection uses job submit timestamp 573 eliminate fixed length strings 574 convert Stork MyString implementations to std::string 575 move Stork job queue log to SPOOL directory 672 user jobs run in LOG directory *SC05 stork matchmaker, tag V6_7-SC2005_Stork_Demo_3-branch (See Nick) *work with Tevfik and grad student Sirish Tummata at LSU to add storage scheduling contacts: Tevfik Kosar <kosar@cct.lsu.edu>, Sirish Tummala <stumma1@lsu.edu> *support stork-announce, stork-discuss lists *start transfers to FNAL tape, from Business School GLOW storage (old) contact: Bill Taylor Installing Stork: See Gnats PR 602 for a description of this documentation request, and the required Stork installation steps. Configuring, Using Stork + HTCondor Credd [Carey's credd, not GreqQ's credd]: Note: all references below are to the "old" condor_credd, aka the Stork credential management daemon. The fact that we now have 2 distinct credential management daemons is a huge source of confusion. Configure a personal stork+credd: # condor_configure --install --make-personal-stork # condor_config_val DAEMON_LIST DAEMON_LIST: COLLECTOR, CREDD, MASTER, NEGOTIATOR, SCHEDD, STARTD, STORK Start HTCondor, including Stork, CredD: # condor_master Using Stork: See tutorial: http://www.cs.wisc.edu/condor/stork/tutorial/stork-dagman/ Using Stork and [Carey's Stork] Credential Manager daemon: Store a X.509 credential in the [Stork] Credential Manager: # stork_store_cred -t x509 -f /tmp/x509up_u19100 -N weber Credential submitted successfully list credentials managed by the [Stork] Credential Manager: # stork_list_cred Name Type ----- ----- weber X509proxy Total 1 Direct Stork to use a credential managed by Credential Manager. Add to stork submit file: cred_name = "weber"; [DO NOT use the x509proxy directive in the stork submit file, when using the cred_name directive. These are exclusive. And no, I haven't yet looked up how a conflict is resolved. Using [Stork] CredD with MyProxy: [see http://grid.ncsa.uiuc.edu/myproxy/credmgmt.html for MyProxy credential mgmt.] [note: the whole condor_credd/myproxy interaction has a working unit test in src/condor_credd/test_credd and src/stork/test_harness_common.sh. run the -myproxy test with the -nomemory and -noleak options.] First, start up a MyProxy server. See http://grid.ncsa.uiuc.edu/myproxy/man/myproxy-server.8.html. The best way to install MyProxy is via the VDT. See Alain. Here's some VDT test script to configure/run the MyProxy server, for testing: --- start myproxy config --- # first be sure to initialize shell with VDT, GLOBUS shell config! # this works best as root if credential dirs are restricted to root export TEMP_DIR=/tmp/myproxy.tmp mkdir -p $TEMP_DIR # Start MyProxy export MYPROXY_PORT=12345 echo "*** Starting MyProxy" cat > $TEMP_DIR/myproxy.conf <<EOF accepted_credentials "*" authorized_retrievers "*" default_retrievers "*" authorized_renewers "*" default_renewers "none" EOF mkdir -p $TEMP_DIR/myproxy_repository chmod 0700 $TEMP_DIR/myproxy_repository $VDT_LOCATION/globus/sbin/myproxy-server -p $MYPROXY_PORT -c $TEMP_DIR/myproxy.conf -s $TEMP_DIR/myproxy_repository --- end myproxy config --- store proxy in myproxy server using myproxy-init: See http://grid.ncsa.uiuc.edu/myproxy/man/myproxy-init.1.html Optionally: review contents of myproxy server, using myproxy-info [http://grid.ncsa.uiuc.edu/myproxy/man/myproxy-info.1.html] Store a credential named 'weber' in the stork credd, from file /tmp/x509up_u19100 which is automatically refreshed from the MyProxy Server username@host:port using the MyProxy distinguished name `grid-cert-info -subject` : # stork_store_cred -t x509 -f /tmp/x509up_u19100 -N weber -m username@host:port -D `grid-cert-info -subject` Now, the condor_credd will check its managed credential every CRED_CHECK_INTERVAL seconds [default=60] and refresh the credential from the copy in the MyProxy server, if needed. The next obvious question is "then why are there _two_ copies of the credential: in both the MyProxy server, and the condor_credd?". There are 2 answers: 1) Carey did it. ;-) This code and reasoning predated Jeff. 2). Jeff can infer that the condor_credd was intended to be a HTCondor daemoncore generic credential management daemon, and was intended to manage a wide variety of credentials, including X.509 proxies, and using the daemoncore API. HTCondor daemoncore clients to the condor_credd only had to retrieve credentials from the [Stork] condor_credd, and did not have to know or care that the condor_credd used MyProxy behind the scenes to refresh X.509 credentials.
