HTCondorWiki: Priv Sep

 
 This page will serve as my brain dump of all things PrivSep. My goal is to highlight things that aren't obvious from looking at the code. In addition, I'm neither dying nor moving farther than 0.2 miles from the Condor project's home in the CS building so please feel free to let me know if something is glaringly missing from this page.
 
-PrivSep is currently implemented for Condor's execute side. The central manager daemons have always been able to run fine without root. The submit side daemons must still run as root unless there is only a single submitter or all submitters are trusted.
+PrivSep is currently implemented for Condor's execute side. The central manager daemons have always been able to run fine without root. The submit side daemons must still run as root unless there is only a single submitter or all submitters are trusted (i.e. a personal Condor).
 
-For a single Condor instance to have both (multi-user) submit-side and execute-side functionality either PrivSep must not be used or the Master must still run as root be configured to start the StartD without root (via the STARTD_USERID setting).
+For a single Condor instance to have both (multi-user) submit-side and execute-side functionality either PrivSep must not be used or the Master must still run as root and be configured to start the StartD without root (via the STARTD_USERID setting).
 
-One other quirk is that PrivSep requres Condor's EXECUTE directory to be "trusted" according to Jim Kupsch's trsuted path algorithm. This means that the EXECUTE directory and all its parent directories must be root-owned and only writable by root. It's a reasonable requirement but it differs from how Condor normally works so it's something to watch out for when moving from non-PrivSep to PrivSep or vice versa. Other files that must be trusted in this same sense include the Switchboard, it's config file (/etc/condor/privsep_config), and the ProcD.
+One other quirk is that PrivSep requires Condor's EXECUTE directory to be "trusted" according to Jim Kupsch's trusted path algorithm. This means that the EXECUTE directory and all its parent directories must be root-owned and only writable by root. It's a reasonable requirement but it differs from how Condor normally works so it's something to watch out for when moving from non-PrivSep to PrivSep or vice versa. Other files that must be trusted in this same sense include the Switchboard, it's config file (/etc/condor/privsep_config), and the ProcD.
 
 {section: Architecture}
 
@@ -18,9 +18,22 @@
 
 The Switchboard is a root-owned setuid program providing the following operations:
 *: Start a ProcD as root
-*: Execute a process as a given user
+*: Execute a program (job) as a given user
 *: Make a directory (needed since the EXECUTE directory is only writable by root)
-*: Recursively chown a directory from/to a given users to/from the Condor user
+*: Recursively chown a directory from/to a given user to/from the Condor user
 *: Recursively remove a directory
 
-In our execute-side PrivSep implementation, most interactions with the root-enabled portion of Condor happen from the Starter. The Master interacts only with the ProcD to manage the process families of its children. The StartD does the same, and additionally uses the Switchboard's recursive remove operation to keep the EXECUTE directory clean. Everything else happens from the Starter.
+In our execute-side PrivSep implementation, most interactions with the root-enabled portion of Condor happen from the Starter. The Master interacts only with the ProcD to manage the process families of its children. The StartD does the same, and additionally uses the Switchboard's recursive remove operation to keep the EXECUTE directory clean.
+
+Everything else happens from the Starter. The workflow looks like this:
+
+*: The Switchboard is used to make the dir_<pid> directory that will be used as the job's sandbox. Unlike non-PrivSep mode, this directory will start out owned by the Condor user.
+*: Condor does everything needed to prepare the job for execution (most notably doing file transfer into the sandbox directory).
+*: Before starting the job, the Switchboard is used to chown the sandbox to the user.
+*: The Switchboard is used to spawn the job as the right UID.
+*: While the job is running, the ProcD is used as usual for process monitoring and control.
+*: When the job is complete, the sandbox is chowned back to the Condor user.
+*: Files can now be transferred out.
+*: Before the Starter exits, the Switchboard is used to clean up the execute directory.
+
+That's pretty much it. One slightly different case concerns VM universe. In that case it is the VMGahp that does the chowns before the job starts and after it completes. The reason is that the VMGahp does its own mucking around with the job sandbox before and after the job runs and the sandbox must be owned by the Condor UID for this to succeed. VM universe also allows for checkpoints which require still more chowning. When a checkpoint happens the VMGahp suspends the VM, the Starter chowns the sandbox to the Condor user, file transfer moves the checkpoint files to the submit machine, the Starter chowns the sandbox back to the user, and finally the VMGahp resumes the job.