This describes the "Transfer Daemon Manager" and schedd functionality to
talk to a transferd.
There are two objects of note, the TransferDaemon object, and the
TDMan object. The schedd creates a single TDMan object which manages
more then one TransferDaemon object. It is ultimately intended that the
TDMan object become its own first class daemon that can be matched
to a schedd for the purposes of handling I/O.
---------------------
TransferDaemon Object
---------------------
The TransferDaemon object is the client interface (but not necessarily
in the sense of a Daemon object) to a running transfer daemon
instance. TransferDaemons are identified by the fully qualified name
of the user for which the transferd was created, an identification
token unique to each transferd and a status which dictates if it in the
process of starting up, ready for work, etc. The object is responsible
for accepting and handling the pushing of TransferRequests to and from
the transferd and handling updates from the transferd. There are certain
callbacks that the schedd will poke into the TransferDaemon object to make
it easier to handle certain state changes of a TransferDaemon. Examples
are when the actual transferd finishes its registration with the schedd,
or when it dies and needs reaping.
An important thing to realize is that the TransferDaemon object is
responsible for calling the callbacks the schedd associated with the
TransferRequests at opportune times when managing the TransferRequests.
Also, note that for each transferd there is a socket connection from it to
the schedd, and from the schedd to it. The former is used to accept the
initial registration and then handle updates of TransferRequests, and the
latter used to push more work to the transferd. This architecture is not
set in stone but was convenient at the time.
Here are the main things that the TransferDaemon object does. I will describe
them as control flows:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
FUNCTION TransferDaemon::add_transfer_request(TransferRequest *treq)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+ Append the transfer request to the end of a list in the TransferDaemon
object of such things.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
FUNCTION TransferDaemon::push_transfer_requests(void)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+ Assert the TransferDaemon has a socket to the actual transferd.
+ Processing the TransferRequests serially:
+ Call the "pre push callback" on the TransferRequest
+ If the returned behavioral control enum was TREQ_ACTION_CONTINUE, then
stop processing the TransferRequest and go on to the next one.
+ If the returned behavioral control enum was TREQ_ACTION_FORGET, then
remove the TransferRequest from future consideration and go on to the
next one.
+ If the returned behavioral control enum was TREQ_ACTION_TERMINATE,
then remove the TransferRequest from future consideration, actually
free it, and move on to the next one.
+ Send the TransferRequest to the transferd. The transferd will create a
"capability" string which represents a token that can be used to talk about
or identify the TransferRequest at a future time. It returns this string
as an attribute in a classad along with another attribute that represents
success/failure of the operation.
+ If the transferd accepted the TransferRequest, then associate the capability
with the TransferRequest in an "in progress" hash table that the
TransferDaemon object is maintaining. Otherwise, poke into the
TransferRequest the reason for failure.
+ Now we call the "post push callback" the schedd had associated with this
TransferRequest and another behavioral control enum is returned.
The current handling of this is unfinished and we assume
TREQ_ACTION_CONTINUE and don't handle what happens if the callback suddenly
wanted to forget about or terminate the TransferRequest while it is
delegated to the transferd itself.
+ end of the function
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
FUNCTION TransferDaemon::update_transfer_request(ClassAd *update)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+ Get the capability string out of the update classad. This capability is
associated with the in progress TransferRequest.
+ If we don't have an in progress TransferRequest then ignore and return.
+ Call the "update" callback on the TransferRequest passing in the update ad.
+ Do a similar behavor to "pre/post push callback" in the above description.
+ Return from the function
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
FUNCTION TransferDaemon::reap_all_transfer_requests(void)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+ Iterate over all non in progress TransferRequest objects.
+ Call the "reaper" call back for the TransferRequest.
+ Assert that the callback returned TREQ_ACTION_TERMINATE, since other
values are meaningless at this time.
+ Free the TransferRequest.
+ Iterate over all in progress TransferRequest objects.
+ Call the "reaper" call back for the TransferRequest.
+ Assert that the callback returned TREQ_ACTION_TERMINATE, since other
values are meaningless at this time.
+ Free the TransferRequest.
------------
TDMan Object
------------
The TDMan object is a Service and manages multiple TransferDaemon objects.
It does a little work during the transferd registration phase to ensure
the right transferd goes with the right user.
Primarily, the TDMan object:
1. Finds/Invokes new transferd instances as needed.
2. Associates the transferd with a user and an identity specific to the
transferd.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
FUNCTION TDMan::invoke_a_td(TransferDaemon *td)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
The purpose of this function is to invoke a transfer daemon and set up the
object passed in with information about the newly created transferd.
+ Assert that the passed in argument isn't null.
+ Now we check the status of the passed in TransferDaemon object.
+ If it is TD_PRE_INVOKED (the usual case), TD_INVOKED (it died after
invocation, but before registering), or TD_MIA (not implemented fully, it
means something else killed the transferd and we noticed at a different
place in the control flow) we keep going.
+ If it is TD_REGISTERED, we return false, since the daemon is already up
and running in a manner we expect.
+ Store the TransferDaemon pointer into a table holding all TransferDaemons.
+ Associate the TransferDaemon object with its own identity.
+ Find the executable we'll be running by paraming for TRANSFERD, except
if it isn't present.
+ Construct the argument list for the transferd, we'll be passing:
-f because it is daemoncore
--schedd <schedd's sinful string>
--id <the id stored in the TransferDaemon object>
--timeout 1200 # twenty minutes to time out
+ Register a TDMan::transferd_reaper() reaper callback for the newly invoked
transferd.
+ Create_Process() the transferd. TODO, currently, this happens as root, but it
needs to happen as the user. This portion of the code hadn't been finished.
+ Keep track of the transferd daemon in a pid table so the reaper knows which
transferd exited when that happens.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
REAPER TDMan::transferd_reaper(long pid, int status)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+ Remove the association of the pid to the TransferDaemon.
+ Remove the association of the user and identity from their respective tables.
+ Call the "reaper" callback on the TransferDaemon and perform whatever
behavor is returned. HOWEVER, we expect TD_ACTION_TERMINATE and anything
else is met with EXCEPT.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
HANDLER TDMan::transferd_registration()
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
The purpose of this function is to figure out what to do when an invoked
transferd comes back and registers itself with the schedd for general use for
that user.
+ Since this is a handler, we are passed a command and a socket.
+ We authenticate the socket and bail if it goes wrong.
+ We get the socket's fully qualified user name.
+ We accept a "registration classad" from the transferd which will contain:
ATTR_TD_SINFUL = <the sinful string of the transferd>
ATTR_TD_ID = <the id the transferd was invoked with on the command line>
+ Now we validate that we actually have a TransferDaemon object associated
with that id and user in the right internal tables. The TransferDaemon
must also be in the TD_INVOKED state. If any of these aren't true,
send back a failure classad containing ATTR_TREQ_INVALID_REQUEST and
ATTR_TREQ_INVALID_REASON and close the connection. Otherwise, send a
classad back with ATTR_TREQ_INVALID_REQUEST set to FALSE.
+ Set the status of the in memory TransferDaemon object to TD_REGISTERED.
+ Store the sinful string of the transferd.
+ Store the socket which we'll use later for updates.
At this point, we need two sockets, one from the transferd to the schedd
(which is the transferd registration socket and in whose handler we now reside)
and one from the schedd back to the transferd, which we now create. It is
VITALLY important that these are non-blocking connections, and that the
control flow on the transferd side has gone back to daemoncore when the schedd
attempts to contact it. Otherwise, we get deadlock.
+ Create a DCTransferD object and call its setup_treq_channel() to get a
socket to the transferd.
+ Store the socket into the TransferDaemon object.
+ We'll be expecting updates from the TransferD, so Register_Socket
TDMan::transferd_update() on the socket we got back from setup_treq_channel().
+ We create a TDUpdateContinuation object which stashes a pile of variables
that we Register_DataPtr() with so the update handler knows for whom it is
working when it is invoked.
+ Now we call any "registration callback" that had been associated with the
TransferDaemon by the schedd. This can do any other work the schedd needs on
behalf of registration. We're going to expect the TD_ACTION_CONTINUE behavior
and EXCEPT otherwise.
+ Push any TransferRequests which had been added to the TransferDaemon.
+ Return KEEP_STREAM.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
SOCKET_HANDLER transferd_update(Stream *sock)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
The purpose of this function is to handle any updates about complete
TransferRequests that the transferd sends back to the schedd.
+ Get the associated DataPtr from daemon core about this handler. This
represents the context of whish TransferDaemon for which this handler
is associated, among other things.
+ Get the "update ad" from the transferd. It must contain:
ATTR_TREQ_CAPABILITY = <a string>
ATTR_TREQ_UPDATE_STATUS = <a string>
ATTR_TREQ_UPDATE_REASON = <a string>
+ Look up the TransferDaemon by its ID.
+ Call td->update_transfer_request() and pass it the update. This function
does the real work with this update ad.
+ return KEEP_STREAM
----------------------
How the Schedd Uses It
----------------------
The schedd cares about the transferd currently in two contexts:
1. When condor_submit wants to submit a job.
This is pretty much implemented and has been known to function.
This control flow may start a transferd.
2. When the schedd wants to start a job.
The control flow is tentatively implemented, but is incomplete.
This control flow may start a transferd.
------
CASE 1
------
The main method by which this occurs is a new handler in the schedd called
Scheduler::requestSandboxLocation(int mode, Stream* s) which is associated
with the REQUEST_SANDBOX_LOCATION command integer.
The high level view of what condor_submit does is:
1. Contact the schedd to get a cluster id & submit job ad.
2. Contact the schedd with the REQUEST_SANDBOX_LOCATION command to have
it locate/start a transferd and give back the sinful string to contact it.
3. Contact the transferd with the TRANSFERD_WRITE_FILES command and
actually upload the files specified in the jobad.
The reason it does three contacts is because this is the best method to
ensure backwards compatibility with previous protocols in addition to easy
indirection and delegation of the "scheduler", "where should my files go",
and "please accept my files, here they are" services that HTCondor provides.
I would do control flow explanations here, but frankly I'm out of time for
documenting this. Therefore:
Check out these functions:
Scheduler::requestSandboxLocation()
The schedd's means by which it starts a transferd and gives the sinful string
back to condor_submit.
DCTransferD::upload_job_files()
The means by which condor_submit actually send the sandbox to a transferd.
And look for 'case STM_USE_TRANSFERD' in condor_submit.V6/submit.cpp
This will tell you how condor_submit actualy interacts with the schedd.
We default the "Sandbox Transfer Method" AKA "STMethod" to
STM_USE_SCHEDD_ONLY for now. Ultimately, this should be changed to
STM_USE_TRANSFERD to enable the transferd functionality on submit.
------
CASE 2
------
The motivation of this code was that the schedd would notice if a job
needed a transferd, and if so, start one up, ensure that it is registered
and ready to go, then start up the transferd, and then the shadow,
telling it what transferd it should use. When the starter is created on the
remote side, the shadow tells it the sinful string of the submit side transferd
and the starter would create the execute side transferd, tell it the
submit side transferd sinful string, and then the transfer would commence.
A converse-like operation would occur when it would be time for the files
to come home. Basically, the starter ask the shadow for a sinful string of a
submit side transferd and then tell the execute side transferd to upload its
files there.
The control flow for this starts in Scheduler:StartJobHandler().
The important part is that if privsep is enabled, and the job needs a
transferd, we then start up a transferd and put the job back into the queue
until the transferd has registered itself.
The functions to be cognizant of are:
Scheduler::jobNeedsTransferd() # returns false, feature wasn't completed.
Scheduler::availableTransferd()
Scheduler::startTransferd()
The CASE 2 portion of the transferd code base is the least developed.
