Page History
In progress.
Author: Alan De Smet (adesmet@cs).
Goal
Be able to run an HTCondor with some nodes only speaking IPv4, some only speaking IPv6, and some speaking both. Matchmaking must avoid matching an IPv4-only computer to an IPv6-only computer.
Assumptions
Many of these are things we would hope to change in the future, but are not required for the initial implementation of mixed-mode.
- Any computer running the condor_collector in such an environment must have both IPv4 and IPv6 connectivity.
- Computers running daemons other than the condor_collector may speak IPv4, IPv6, or both.
- Only the vanilla universe is required to work.
- It is not necessary to support listening on more than one IPv4 and one IPv6 address for a given host. This mirrors today's limitation that only one IPv6 address can be used by a given host.
- It is acceptable to require computers speaking IPv6 to specify which address to use, including for IPv4 if the computer also speaks that. This mirrors today's limitation that IPv6 hosts must specify the address to use.
- We can ignore the complexities of private and link local addresses.
- It is acceptable for a computer that only speaks IPv4/IPv6 to be unable to communicate in any way with a computer that only speaks IPv6/IPv4. For example, from an IPv6-only computer, "condor_q -name 123.45.67.89" is allowed to fail.
Open design questions
- Do any of these things work in IPv6? If not, identify them as assumptions to not work above.
- BIND_ALL_INTERFACES=TRUE
- ENABLE_ADDRESS_REWRITING=TRUE
- Asterisks in NETWORK_INTERFACE
- Host and IP based security
- How to specify COLLECTOR_HOST? Do the various querying mechanisms actually try a list, or can we only broadcast to multiple (via CONDOR_VIEW_HOST). A very cursory look suggests lots of places are trying to parse COLLECTOR_HOST on their own; we'll probably need to unify this.
- What are we passing in on connect attempts? sinful string or object equivalent?
Plan
- Dual mode collector
- Teach collector to respond to commands on both IPv4 and IPv6. (Likely side effect: all daemons do.)
- Teach collector to hold both IPv4 and IPv6 ads. (Probably no work at all.)
- Dual address sinful strings
- Allow encoding both IPv4 and IPv6 addresses in a sinful string. Needs to simultaneously work with CCB and shared port. If we bother with backward compatbility, make the IPv4 address the backward compatible one.
- May need to support different port numbers for different addresses; trying to synchronize port numbers across TCP and UDP for both IPv4 and IPv6 seems like a pain in the ass.
- Design such that if in the future we wanted to have arbitrarily long lists of addresses, we could do so while maintaining backward compatbility (older versions would just use the head of a list). This seems likely to be functionality we'll want in the future, and while we're messing with the sinful strings in the time to ensure the futureproofing is done.
- Rejected: MyAddress (implicitly IPv4) and new attribute MyAddressIPv6. It means multiple places need to look in multiple attributes to find the appropriate address. We want to be able to hand a sinful string around and assume it's everything you need to contact someone. We made this same tradeoff of CCB and shared port.
- Teach startd to listen on and advertise both IPv4 and IPv6 addresses. (Likely side effect: all daemons do.)
- Likely just done at the DaemonCore level where the attributes are automatically added. Indeed, likely even lower, at the "Gimme a sinful string for myself" level.
- There will be breakage at this point, as the negotiator begins trying to match IPv4-only hosts to IPv6-only hosts.
- Teach schedd to listen on and advertise both IPv4 and IPv6 addresses. (Possibly done by "teach startd" step.)
- There will be breakage at this point, as the negotiator begins trying to match IPv4-only hosts to IPv6-only hosts.
- Teach negotiator to make matchmaking decisions based on IPv4 and IPv6 capabilities of the involved parties.
- New function: CanCommunicate, takes two sinful strings, returns true or false.
- Negotiator implicitly adds CanCommunicate to Requirements for the Machine (or the Job; whichever).
- Rejected: Explicitly add. Risks trashing autoclusters, as every machine and job has a unique sinful string.
- Rejected: add SpeaksIPv6=TRUE; SpeaksIPv4=TRUE then adding to Job/Machine adds (Requirements=SpeaksIPv6 && TARGET.SpeaksIPv6)||(Requirements=SpeaksIPv4 && TARGET.SpeaksIPv4). Not backward compatible. Could assume anyone lacking both only speaks IPv4, but breaks existance IPv6 pools. Not future proof: no hope for subtle things like "are these hosts on the same subnet and thus able to use link local IPv6?"
- Teach other daemons to listen on and advertise both IPv4 and IPv6 addresses. (Possibly done by "teach startds" step).
- Gracefully handle impossible connection requests.
- Daemons that only handle one protocol shouldn't ever have a reason to connect to the other. But just in case, ensure the failure is graceful: complain to the log and carry on. EXCEPTing is absolutely not acceptable.
- Command line tools could easily be asked to do the impossible. "condor_q -name ipv6-only-host.example.com" on an IPv4 host, for example. We need to give a reasonable message to the end user.