By Greg Quinn

Introduction

This page will serve as my brain dump of all things PrivSep. My goal is to highlight things that aren't obvious from looking at the code. In addition, I'm neither dying nor moving farther than 0.2 miles from the HTCondor project's home in the CS building so please feel free to let me know if something is glaringly missing from this page.

PrivSep is currently implemented for HTCondor's execute side. The central manager daemons have always been able to run fine without root. The submit side daemons must still run as root unless there is only a single submitter or all submitters are trusted (i.e. a personal HTCondor).

For a single HTCondor instance to have both (multi-user) submit-side and execute-side functionality either PrivSep must not be used or the Master must still run as root and be configured to start the StartD without root (via the STARTD_USERID setting).

One other quirk is that PrivSep requires HTCondor's EXECUTE directory to be "trusted" according to Jim Kupsch's trusted path algorithm. This means that the EXECUTE directory and all its parent directories must be root-owned and only writable by root. It's a reasonable requirement but it differs from how HTCondor normally works so it's something to watch out for when moving from non-PrivSep to PrivSep or vice versa. Other files that must be trusted in this same sense include the Switchboard, it's config file (/etc/condor/privsep_config), and the ProcD.

Architecture

PrivSep's architecture is pretty simple. Only a small portion of the actual HTCondor code requires root privileges. For now, that includes the condor_root_switchboard and condor_procd programs. (See #110 on how this could change to only include the Switchboard.) The rest of execute-side HTCondor runs as an unprivileged user and makes requests of the root-enabled components when necessary. It is the responsibility of the root-enabled pieces to only honor "valid" requests from other HTCondor daemons. The policy as to what is valid is determined partly by the Switchboard's configuration file, which lists users and directories over which HTCondor should be allowed to operate.

The ProcD provides operations relevant to tracking, monitoring, and controlling processes and is described in more detail on the ProcdWisdom page.

The Switchboard is a root-owned setuid program providing the following operations:

In our execute-side PrivSep implementation, most interactions with the root-enabled portion of HTCondor happen from the Starter. The Master interacts only with the ProcD to manage the process families of its children. The StartD does the same, and additionally uses the Switchboard's recursive remove operation to keep the EXECUTE directory clean.

Everything else happens from the Starter. The workflow looks like this:

That's pretty much it. One slightly different case concerns VM universe. In that case it is the VMGahp that does the chowns before the job starts and after it completes. The reason is that the VMGahp does its own mucking around with the job sandbox before and after the job runs and the sandbox must be owned by the HTCondor UID for this to succeed. VM universe also allows for checkpoints which require still more chowning. When a checkpoint happens the VMGahp suspends the VM, the Starter chowns the sandbox to the HTCondor user, file transfer moves the checkpoint files to the submit machine, the Starter chowns the sandbox back to the user, and finally the VMGahp resumes the job.

glexec

See the GlexecWisdom page for information on how HTCondor can use glexec to provide functionality similar to PrivSep.