{verbatim} This describes the "Transfer Daemon Manager" and schedd functionality to talk to a transferd. There are two objects of note, the TransferDaemon object, and the TDMan object. The schedd creates a single TDMan object which manages more then one TransferDaemon object. It is ultimately intended that the TDMan object become its own first class daemon that can be matched to a schedd for the purposes of handling I/O. --------------------- TransferDaemon Object --------------------- The TransferDaemon object is the client interface (but not necessarily in the sense of a Daemon object) to a running transfer daemon instance. TransferDaemons are identified by the fully qualified name of the user for which the transferd was created, an identification token unique to each transferd and a status which dictates if it in the process of starting up, ready for work, etc. The object is responsible for accepting and handling the pushing of TransferRequests to and from the transferd and handling updates from the transferd. There are certain callbacks that the schedd will poke into the TransferDaemon object to make it easier to handle certain state changes of a TransferDaemon. Examples are when the actual transferd finishes its registration with the schedd, or when it dies and needs reaping. An important thing to realize is that the TransferDaemon object is responsible for calling the callbacks the schedd associated with the TransferRequests at opportune times when managing the TransferRequests. Also, note that for each transferd there is a socket connection from it to the schedd, and from the schedd to it. The former is used to accept the initial registration and then handle updates of TransferRequests, and the latter used to push more work to the transferd. This architecture is not set in stone but was convenient at the time. Here are the main things that the TransferDaemon object does. I will describe them as control flows: ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; FUNCTION TransferDaemon::add_transfer_request(TransferRequest *treq) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + Append the transfer request to the end of a list in the TransferDaemon object of such things. ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; FUNCTION TransferDaemon::push_transfer_requests(void) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + Assert the TransferDaemon has a socket to the actual transferd. + Processing the TransferRequests serially: + Call the "pre push callback" on the TransferRequest + If the returned behavioral control enum was TREQ_ACTION_CONTINUE, then stop processing the TransferRequest and go on to the next one. + If the returned behavioral control enum was TREQ_ACTION_FORGET, then remove the TransferRequest from future consideration and go on to the next one. + If the returned behavioral control enum was TREQ_ACTION_TERMINATE, then remove the TransferRequest from future consideration, actually free it, and move on to the next one. + Send the TransferRequest to the transferd. The transferd will create a "capability" string which represents a token that can be used to talk about or identify the TransferRequest at a future time. It returns this string as an attribute in a classad along with another attribute that represents success/failure of the operation. + If the transferd accepted the TransferRequest, then associate the capability with the TransferRequest in an "in progress" hash table that the TransferDaemon object is maintaining. Otherwise, poke into the TransferRequest the reason for failure. + Now we call the "post push callback" the schedd had associated with this TransferRequest and another behavioral control enum is returned. The current handling of this is unfinished and we assume TREQ_ACTION_CONTINUE and don't handle what happens if the callback suddenly wanted to forget about or terminate the TransferRequest while it is delegated to the transferd itself. + end of the function ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; FUNCTION TransferDaemon::update_transfer_request(ClassAd *update) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + Get the capability string out of the update classad. This capability is associated with the in progress TransferRequest. + If we don't have an in progress TransferRequest then ignore and return. + Call the "update" callback on the TransferRequest passing in the update ad. + Do a similar behavor to "pre/post push callback" in the above description. + Return from the function ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; FUNCTION TransferDaemon::reap_all_transfer_requests(void) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + Iterate over all non in progress TransferRequest objects. + Call the "reaper" call back for the TransferRequest. + Assert that the callback returned TREQ_ACTION_TERMINATE, since other values are meaningless at this time. + Free the TransferRequest. + Iterate over all in progress TransferRequest objects. + Call the "reaper" call back for the TransferRequest. + Assert that the callback returned TREQ_ACTION_TERMINATE, since other values are meaningless at this time. + Free the TransferRequest. ------------ TDMan Object ------------ The TDMan object is a Service and manages multiple TransferDaemon objects. It does a little work during the transferd registration phase to ensure the right transferd goes with the right user. Primarily, the TDMan object: 1. Finds/Invokes new transferd instances as needed. 2. Associates the transferd with a user and an identity specific to the transferd. ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; FUNCTION TDMan::invoke_a_td(TransferDaemon *td) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; The purpose of this function is to invoke a transfer daemon and set up the object passed in with information about the newly created transferd. + Assert that the passed in argument isn't null. + Now we check the status of the passed in TransferDaemon object. + If it is TD_PRE_INVOKED (the usual case), TD_INVOKED (it died after invocation, but before registering), or TD_MIA (not implemented fully, it means something else killed the transferd and we noticed at a different place in the control flow) we keep going. + If it is TD_REGISTERED, we return false, since the daemon is already up and running in a manner we expect. + Store the TransferDaemon pointer into a table holding all TransferDaemons. + Associate the TransferDaemon object with its own identity. + Find the executable we'll be running by paraming for TRANSFERD, except if it isn't present. + Construct the argument list for the transferd, we'll be passing: -f because it is daemoncore --schedd --id --timeout 1200 # twenty minutes to time out + Register a TDMan::transferd_reaper() reaper callback for the newly invoked transferd. + Create_Process() the transferd. TODO, currently, this happens as root, but it needs to happen as the user. This portion of the code hadn't been finished. + Keep track of the transferd daemon in a pid table so the reaper knows which transferd exited when that happens. ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; REAPER TDMan::transferd_reaper(long pid, int status) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + Remove the association of the pid to the TransferDaemon. + Remove the association of the user and identity from their respective tables. + Call the "reaper" callback on the TransferDaemon and perform whatever behavor is returned. HOWEVER, we expect TD_ACTION_TERMINATE and anything else is met with EXCEPT. ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; HANDLER TDMan::transferd_registration() ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; The purpose of this function is to figure out what to do when an invoked transferd comes back and registers itself with the schedd for general use for that user. + Since this is a handler, we are passed a command and a socket. + We authenticate the socket and bail if it goes wrong. + We get the socket's fully qualified user name. + We accept a "registration classad" from the transferd which will contain: ATTR_TD_SINFUL = ATTR_TD_ID = + Now we validate that we actually have a TransferDaemon object associated with that id and user in the right internal tables. The TransferDaemon must also be in the TD_INVOKED state. If any of these aren't true, send back a failure classad containing ATTR_TREQ_INVALID_REQUEST and ATTR_TREQ_INVALID_REASON and close the connection. Otherwise, send a classad back with ATTR_TREQ_INVALID_REQUEST set to FALSE. + Set the status of the in memory TransferDaemon object to TD_REGISTERED. + Store the sinful string of the transferd. + Store the socket which we'll use later for updates. At this point, we need two sockets, one from the transferd to the schedd (which is the transferd registration socket and in whose handler we now reside) and one from the schedd back to the transferd, which we now create. It is VITALLY important that these are non-blocking connections, and that the control flow on the transferd side has gone back to daemoncore when the schedd attempts to contact it. Otherwise, we get deadlock. + Create a DCTransferD object and call its setup_treq_channel() to get a socket to the transferd. + Store the socket into the TransferDaemon object. + We'll be expecting updates from the TransferD, so Register_Socket TDMan::transferd_update() on the socket we got back from setup_treq_channel(). + We create a TDUpdateContinuation object which stashes a pile of variables that we Register_DataPtr() with so the update handler knows for whom it is working when it is invoked. + Now we call any "registration callback" that had been associated with the TransferDaemon by the schedd. This can do any other work the schedd needs on behalf of registration. We're going to expect the TD_ACTION_CONTINUE behavior and EXCEPT otherwise. + Push any TransferRequests which had been added to the TransferDaemon. + Return KEEP_STREAM. ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; SOCKET_HANDLER transferd_update(Stream *sock) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; The purpose of this function is to handle any updates about complete TransferRequests that the transferd sends back to the schedd. + Get the associated DataPtr from daemon core about this handler. This represents the context of whish TransferDaemon for which this handler is associated, among other things. + Get the "update ad" from the transferd. It must contain: ATTR_TREQ_CAPABILITY = ATTR_TREQ_UPDATE_STATUS = ATTR_TREQ_UPDATE_REASON = + Look up the TransferDaemon by its ID. + Call td->update_transfer_request() and pass it the update. This function does the real work with this update ad. + return KEEP_STREAM ---------------------- How the Schedd Uses It ---------------------- The schedd cares about the transferd currently in two contexts: 1. When condor_submit wants to submit a job. This is pretty much implemented and has been known to function. This control flow may start a transferd. 2. When the schedd wants to start a job. The control flow is tentatively implemented, but is incomplete. This control flow may start a transferd. ------ CASE 1 ------ The main method by which this occurs is a new handler in the schedd called Scheduler::requestSandboxLocation(int mode, Stream* s) which is associated with the REQUEST_SANDBOX_LOCATION command integer. The high level view of what condor_submit does is: 1. Contact the schedd to get a cluster id & submit job ad. 2. Contact the schedd with the REQUEST_SANDBOX_LOCATION command to have it locate/start a transferd and give back the sinful string to contact it. 3. Contact the transferd with the TRANSFERD_WRITE_FILES command and actually upload the files specified in the jobad. The reason it does three contacts is because this is the best method to ensure backwards compatibility with previous protocols in addition to easy indirection and delegation of the "scheduler", "where should my files go", and "please accept my files, here they are" services that HTCondor provides. I would do control flow explanations here, but frankly I'm out of time for documenting this. Therefore: Check out these functions: Scheduler::requestSandboxLocation() The schedd's means by which it starts a transferd and gives the sinful string back to condor_submit. DCTransferD::upload_job_files() The means by which condor_submit actually send the sandbox to a transferd. And look for 'case STM_USE_TRANSFERD' in condor_submit.V6/submit.cpp This will tell you how condor_submit actualy interacts with the schedd. We default the "Sandbox Transfer Method" AKA "STMethod" to STM_USE_SCHEDD_ONLY for now. Ultimately, this should be changed to STM_USE_TRANSFERD to enable the transferd functionality on submit. ------ CASE 2 ------ The motivation of this code was that the schedd would notice if a job needed a transferd, and if so, start one up, ensure that it is registered and ready to go, then start up the transferd, and then the shadow, telling it what transferd it should use. When the starter is created on the remote side, the shadow tells it the sinful string of the submit side transferd and the starter would create the execute side transferd, tell it the submit side transferd sinful string, and then the transfer would commence. A converse-like operation would occur when it would be time for the files to come home. Basically, the starter ask the shadow for a sinful string of a submit side transferd and then tell the execute side transferd to upload its files there. The control flow for this starts in Scheduler:StartJobHandler(). The important part is that if privsep is enabled, and the job needs a transferd, we then start up a transferd and put the job back into the queue until the transferd has registered itself. The functions to be cognizant of are: Scheduler::jobNeedsTransferd() # returns false, feature wasn't completed. Scheduler::availableTransferd() Scheduler::startTransferd() The CASE 2 portion of the transferd code base is the least developed. {endverbatim}