{section:Interactive Singularity Jobs and condor_ssh_to_job}
Starting with HTCondor 8.8, =condor_ssh_to_job= and hence also interactive jobs use an =sshd= running directly on the execute node with user privileges. Since 8.8.10, most issue have been ironed out, and
connecting into the job happens using =condor_nsenter=, which is an =nsenter= -like tool to "enter" container namespaces in a generic way. This tool is spawned by the =starter= in parallel to =sshd=.

There are a few remaining issues related to X11 forwarding which can be worked around, and which are partially dependent on the utilised setup. These are discussed on this page.

{subsection:X11 forwarding}
X11 forwarding in general works by running =xauth= as a child of the =sshd= process on the execute node. =sshd= mostly prunes the environment before, setting a new DISPLAY variable to a forwarded X11 port. It then runs =xauth= which by default uses the user's home directory to store the X11 authorization information.

Two issues arise:
*: Since =condor_nsenter= does not run as a child of the =sshd=, but as a child of the =starter=, it can not pass on the =DISPLAY= environment variable to the user session.
*: In many cases, when containers are used, the actual users may not have a home directory on the execute node, or might not have it mounted inside the container. However, we cannot override the location to store the =.Xauthority= file with the environment variable =XAUTHORITY= since =sshd= prunes that.

{subsubsection:A possible workaround}
To solve both issues at once, we can make use of the fact that =sshd= is spawned and configured by HTCondor via the =condor_ssh_to_job_sshd_config_template=. The location of this template can be set via the knob =SSH_TO_JOB_SSHD_CONFIG_TEMPLATE=.
We can patch the file shipped with HTCondor and add the line:
  XAuthLocation /usr/local/bin/condor_xauth_wrapper
Subsequently, we can create the wrapper script (make sure it is executable) with the following content:
----
{verbatim}
#!/bin/bash

# Walk up the process tree until we find the second sshd which rewrites cmdline to "sshd: user@tty".
# The first sshd is our parent process which does not log itself.
SSHD_PID=$$
SSHD_CNT=0
while true; do
  IFS= read -r -d '' CMDLINE </proc/${SSHD_PID}/cmdline || [[ $cmdline ]]
  #echo "Checking ID ${SSHD_PID}, cmdline ${CMDLINE^^}"
  SSHD_MATCHER="^SSHD: "
  if [[ ${CMDLINE^^} =~ ${SSHD_MATCHER} ]]; then
    # We found the sshd!
    SSHD_CNT=$(( SSHD_CNT + 1))
    if [ ${SSHD_CNT} -gt 1 ]; then
      break;
    fi
  fi
  SSHD_PID=$(ps -o ppid= -p ${SSHD_PID} | awk '{print $1}')
  if [ ${SSHD_PID} -eq 1 ]; then
    # We arrived at the INIT process, something very wrong... Let's stop and alert the user.
    echo "Error: Could not determine sshd process, X11 forwarding will not work!"
    echo "       Please let your admins know you got this error!"
    exit 0
  fi
done
#echo "SSHD PID is ${SSHD_PID}."

# Find sshd.log, checking through fds.
FOUND_SSHD_LOG=0
for FD in $(ls -1 /proc/${SSHD_PID}/fd/); do
  FILE=$(readlink -f /proc/${SSHD_PID}/fd/$FD)
  #echo "Checking FD $FD, file is $FILE"
  SSHD_LOG_MATCHER="sshd\.log$"
  if [[ "${FILE}" =~ ${SSHD_LOG_MATCHER} ]]; then
    #echo "Found ${FILE}!"
    FOUND_SSHD_LOG=1
    SSH_TO_JOB_DIR=$(dirname ${FILE})
    JOB_WORKING_DIR=$(dirname ${SSH_TO_JOB_DIR})
    break;
  fi
done

if [ ${FOUND_SSHD_LOG} -eq 0 ]; then
  # We could not identify sshd.log, let's stop and alert the user.
  echo "Error: Could not determine sshd process' (PID: ${SSHD_PID}) log, X11 forwarding will not work!"
  echo "       Please let your admins know you got this error!"
  exit 0
fi

# Finally, if we arrive here, all is well.

# This does NOT work, since env.sh is sourced as forced command, too early.
#echo "export DISPLAY=${DISPLAY}" >> ${SSH_TO_JOB_DIR}/env.sh

# Ugly hack needed with HTCondor 8.8.10 which does not yet pass through DISPLAY.
echo "export DISPLAY=${DISPLAY}" > ${JOB_WORKING_DIR}/.display

export XAUTHORITY=${JOB_WORKING_DIR}/.Xauthority
/usr/bin/xauth "$@" </dev/stdin
{endverbatim}
----
Please note that this script is pretty verbose, and handles very unlikely errors not observed in practice (yet). Most of the code is just there to find out which directory is used as the execute directory for the job, then place the =DISPLAY= environment variable inside a file =.display= in there, and finally adjust the environment variable =XAUTHORITY= to place the =.Xauthority= file there.

This script works combined with two environment hack inside the container:
*: The container needs to use the execute directory as =HOME= directory. You can for example do that by setting =SINGULARITY_HOME= in the =STARTER_JOB_ENVIRONMENT= knob, which you may likely touch anyways to set =SINGULARITY_NOHOME=true= if you run without the users' home directories on the execute node.
*: The container needs to source =.display= from the =HOME= directory on start (and may also clean up by deleting it).
Example code for the latter task could be, assuming your in-container =HOME= directory is =/jwd=:
  if [ -r /jwd/.display ]; then
        source /jwd/.display
        rm -f /jwd/.display
  fi
A good place could be a file in =/etc/profile= inside the container(s) you use.

{subsubsection:An alternative workaround}
Note that if your =sshd= is recent enough and understands =SetEnv= (should be the case starting from versions >=7.8), you could also patch =/usr/libexec/condor/condor_ssh_to_job_sshd_setup= instead and inject the =XAUTHORITY= location via =SetEnv=. In that script, =${base_dir}= refers to the execute directory. This solution has not yet been tested, but should also achieve the expected result.
However, you will still need the above =condor_xauth_wrapper= for now, to transport over the =DISPLAY= variable by hooking into =sshd=. But as soon as HTCondor learns this in a future version, the lengthy script can be dropped.