Page History

Turn Off History

How to get HTCondor and a NAT Firewall to Cooperate

For an HTCondor machine on which you want to do useful work either as a submit node or as an execute node, you also would like to share your new computer resources with others outside your department or division. But, you do not want crooks using your systems to wreak havoc. So, you install a firewall between your HTCondor resource and the Internet. In this description, we assume that the firewall is a separate host from the HTCondor resource. We further assume explicitly that the firewall is a bastion host running Linux, and the firewall is iptables, with Network Address Translation (NAT); this assumption allows the description to contain explicit commands to run. You or your firewall administrator should be able to translate these instructions as needed to your particular firewall installation. We are also assuming that you are not going to be using CCB.

Assume that HTCondor is installed and running with the following set up:

The submit machine, which runs the condor_schedd is installed on a machine named S with IP address 192.168.0.1.

The execute machine, which runs the condor_startd is installed on a machine named E with IP address 192.168.0.2.

The firewall has an Internet-facing, external IP address of 10.0.0.1. Its internal, ---that is, facing toward your local network---IP address of 192.168.0.250; we will call this machine F. We know 10.0.0.1 is actually not a routable address, but pretend that it is for the duration of this document. S and E are in the domain mydomain.net.

Then we will make the following changes to condor_config.local on S (the schedd). To find your HTCondor configuration files, the command condor_config_val -dump will be a big help, as the files are listed in the header of the output 

USE_SHARED_PORT = True
SHARED_PORT_ARGS = -p 9617
PRIVATE_NETWORK_NAME = mydomain.net
PRIVATE_NETWORK_INTERFACE = eth0
TCP_FORWARDING_HOST = 10.0.0.1
In the configuration settings above, the port 9617 was chosen out of a hat; there is no reason it cannot be any port on the system. 9618 is often chosen; it is the well-known port of the HTCondor collector. In our setup, we are not assuming that there is a collector in the 192.168.0.0/24 network that will be contacted from outside 192.168.0.0/24, so 9618 is also a valid port number; but you may well want to avoid 9618 if you have an internal collector. Note that the TCP_FORWARDING_HOST must match the external address of the collector.

On the execute node E, we have similar configuration changes, except for the shared port: 

USE_SHARED_PORT = True
SHARED_PORT_ARGS = -p 9616
PRIVATE_NETWORK_NAME = mydomain.net
PRIVATE_NETWORK_INTERFACE = eth0
TCP_FORWARDING_HOST = 10.0.0.1
The use of PRIVATE_NETWORK_NAME on S and E allow them to communicate directly without going through the firewall F.

Now, on the firewall F, we run the following commands to redirect connections from the Internet to ports 9617 and 9616 on F to the corresponding ports on S and E: 

iptables -t nat -A PREROUTING -p tcp -d 10.0.0.1 --dport 9617 -j DNAT --to-destination 192.168.0.1
iptables -t nat -A PREROUTING -p tcp -d 10.0.0.1 --dport 9616 -j DNAT --to-destination 192.168.0.2
iptables -A POSTROUTING -o eth1 -j SNAT --to-source 10.0.0.1
The first rule above says that inbound connections to 10.0.0.1 on port 9617 will be rewritten to be connections to 192.168.0.1, port 9617. The second line states a similar rule for 192.168.0.2, port 9616. The third rule is probably superfluous, as it is likely already in the firewall rules: all outbound connections from 10.0.0.1 will look like they emanate from 10.0.0.1.

Finally, run condor_reconfig on S and E to incorporate the configuration changes.

One may well ask how this works for HTCondor. Simply put, HTCondor wraps all the information in a "sinful string". condor_status -schedd -l will return a Schedd ClassAd, which will contain a line that looks something like: 

MyAddress = "<10.0.0.1:9617?PrivAddr=%3c192.168.0.1:9617%3fsock%3d936_480b_8%3e&PrivNet=mydomain.net&noUDP&sock=936_480b_8>"
This string contains the needed information for another HTCondor client or daemon to contact the schedd and begin using high throughput computing.