See also: *: IpvSixWorkNoteTwo *: IpvSixWorkNoteThree Maintainer: MinJae Hwang(m@cs) Comment: Use ticket #9 {section: These pages are outdated outrageously. Do not believe anything written here} {section: Introduction} This page is work note of converting HTCondor to IPv6 compatible. More precisely, IPv4/IPv6 both compatible. {section: What should be changed?} All the deprecated BSD socket functions. These are complete list. (from {link: http://people.redhat.com/drepper/userapi-ipv6.html Userlevel IPv6 Programming Introduction}) {code} struct sockaddr_in gethostbyname gethostbyaddr getservbyaddr inet_addr inet_aton inet_ntoa {endcode} {section: note} gethostbyname() returns DNS aliases for host name. I could not find way to retrieve DNS aliases using getaddrinfo(). Only 2 location uses aliases field. getservbyaddr() returns struct servent, which is also deprecated data structure. Only 2 location calls getservbyaddr() and they only use a port number field of servent. So, I would make wrapper function that only returns a port number. {section: ...} HTCondor's utility function that returns or accepts sockaddr_in directly. These functions will be converted if we change top-level functions (listed above). {code} string_to_sin : 28 address_to_sin : 1 sin_to_ipstring : 2 sock_peer_to_string : 1 sin_to_hostname : 15 display_from : 0 is_ipaddr_implementation : 2 is_valid_network : 13 string_to_ip : 16 getSockAddr : 9 {endcode} IP address is no longer fixed 4 bytes nor fixed 16 bytes. It can be both of them. So, every storage class should be changed. For some of source codes that use 'int' as storage for IP address, this is most troublesome because it could be hidden to simple text search. Reading and parsing IP address from config file should be changed as well. If an existing code base entirely relied on BSD socket interface such as inet_aton or inet_addr, it would be easier. However, if the code has proprietary parsing/converting functions, every incident should be found and changed. Zach mentioned that security and authentication code mangled with IPv4.(IpVerifier) IPv6 address string is far longer than IPv4 address. IPv6 address [0001:0002:0003:0004:0005:0006:0007:0008], length is 41. IPv4 address 101.202.103.203, length is 15. About 3 times longer. Printing IP address to log file should be considered since existing layout may not work well. {section: Current HTCondor Code Base} As noted earlier, HTCondor source has 576 lines that uses obsolete BSD functions. What makes problem complicated is that unsigned long is used to hold IP address. There are many places to change. For example, in sock.h,{linebreak} {code} unsigned int Sock::get_ip_int() unsigned int Sock::peer_ip_int() {endcode} It uses 'unsigned int' to pass IP address. class Sock already has abstraction of BSD socket interface. But, IP address escapes from this class by returning it as an 4 byte int and sockaddr_in. It is required to check whether there are codes that directly calls BSD socket interface. {section: The Method of Attack} Having single IP address class that deals both IPv4 and IPv6. So, nobody uses sockaddr, sockaddr_in, or plain int to represent IP address. Extend existing Sock class to be able to deal with both IPv4 and IPv6. Extend sinful string handler to be able to deal with both IPv4 and IPv6 address. Here is {link: http://www.boost.org/doc/libs/1_41_0/boost/asio/ip/address.hpp Boost IP address class}. In that class, they hold both IPv4 address and IPv6 address. But I think we only need one 16 byte array instead of having both. Also, we may not need to have ipv4/v6 flag because there is IPv4 address mapping in IPv6. [Zach commented that although we do not need separate storage for each IPv4 and IPv6 address, we still need a type because when we tries to connect to other machines, we do not know whether it should be done through IPv4 network or IPv6 network] Change every IP address storage to that IP address class. It is not decided whether to have host byte order or network byte order. sockaddr_in always have network byte order in it. However, the class itself should be convertible to sockaddr_in or sockaddr_in6. However, byte-level compatibility may not be required. So, we need something like IPaddress::to_sockaddr_in6() function. {section: Unified Address Class} I juggled many design possibilities. I came to the conclusion that unified address class should contain sockaddr_in6. Here is how sockaddr_in6 look like {code} // code from struct sockaddr_in6 { unsigned short int sin6_family; /* AF_INET6 */ __u16 sin6_port; /* Transport layer port # */ __u32 sin6_flowinfo; /* IPv6 flow information */ struct in6_addr sin6_addr; /* IPv6 address */ __u32 sin6_scope_id; /* scope id (new in RFC2553) */ }; {endcode} Notable difference from sockaddr_in(IPv4 address/port storage) is that it has scope id. Scope id denotes a ethernet interface in a system. Scope id binds IP address shown in sockaddr_in6 to specific ethernet interface. That binding is necessary because some IPv6 addresses are limited to some ethernet interface. Google about 'link local address' would be helpful. Tentative unified address class is as follows {code} class ipaddr { sockaddr_in6 storage; const unsigned char* get_addr() const; unsigned char* get_addr(); unsigned int& get_addr_header(); const unsigned int& get_v4_addr() const; unsigned int& get_v4_addr(); public: // accepts network byte ordered ip and port void init(int ip, unsigned port); ipaddr(); // ip address is always network byte order but port is not. ipaddr(int ip, unsigned port = 0); ipaddr(in_addr ip, unsigned port = 0); ipaddr(const in6_addr& in6, unsigned port = 0); ipaddr(sockaddr_in* sin) ; ipaddr(sockaddr_in6* sin6); sockaddr_in to_sin() const; const sockaddr_in6& to_sin6() const; bool is_ipv4() const; }; {endcode} As you see here, sockaddr_in6 is super-set of sockaddr_in. So, this implementation considers IPv6 as basis and providing compatibility to IPv4 usage. There should be issue when a compiler does not have IPv6 related header files. In that case, we may supply definition of sockaddr_in6. Here only dependency is to AF_INET6 constant and struct sockaddr_in6. In IPv6, you cannot simply have 16byte IP address portion. You should have scope id as well. Separating { IP address, scope id } and port number could be possible but it will only make source codes complex. Some might argue that in this way, we could save some memory foot-print but I might say that considering size of sockaddr_in6, it is no big deal. In bottom line, I recommend you to have class IpAddr (which is basically sockaddr_in6 with helper functions) even you only want IP address. {section: Make configure for IPv6 support} When creating sockets and binding to local address, you need to specify whether it is IPv4 or IPv6. To supply these parameters, I am considering to use ifdef for C/C++ source codes. {code} #ifdef CONDOR_IPV6_SUPPORT fd = socket(AF_INET6, ...); #else fd = socket(AF_INET,...); #endif {endcode} Let's find a way to configure it. It may be done by using "configure" script. {section: HTCondor NO_DNS option} In HTCondor, NO_DNS option eliminates its dependency to Domain Name System. According to condor_netdb.c {code} /* SPECIAL NAMES: * * If NO_DNS is configured we do some special name/ip handling. IP * addresses given to gethostbyaddr, XXX.XXX.XXX.XXX, will return * XXX-XXX-XXX-XXX.DEFAULT_DOMAIN, and an attempt will be made to * parse gethostbyname names from XXX-XXX-XXX-XXX.DEFAULT_DOMAIN into * XXX.XXX.XXX.XXX. */ {endcode} Its name mangling rule is restricted to IPv4. So it should be extended to IPv6 as well. There are restrictions on host name in DNS. According to the Wikipedia, Host names are restricted to a small subset of the ASCII character set known as LDH, the Letters A-Z in upper and lower case, Digits 0–9, Hyphen, and the dot to separate LDH-labels; see RFC 3696 section 2 for details. This prevented the representation of names and words of many languages natively. Thus, we may convert IPv6 address [aaaa:bbbb:cccc:dddd:eeee:ffff:0000:1111] to aaaa-bbbb-cccc-dddd-eeee-ffff-0000-1111. {section: Converting obsolete BSD interface} Since we have few hundreds lines of code, I think I can change it by manually. But scary thing is how to test all of them. Do you have any idea? I am desperate. {section: Changing every lines which dealing with IP address} My plan is to use compiler generated error. This plan has some analogy to 'escape analysis'. Consider "old" IP address structure is escaping from socket interfaces to every leaf of source codes. If we disallow from the top, there will be no IPv4 address flowing to the leaf. Here is flow. 1:Find every BSD socket functions that returns IP address or accepts IP address argument 1:Make proxy function for all of them. 1:Change every lines where calls old functions to proxy functions. Definitely, proxy functions should allow only new class "IpAddr" instead of sockaddr, and sockaddr_in. 1:You will have tons of error message. 1:Resolving this all error message with class IpAddr. 1:You have the code base that compatible with IPv6 sockets. {section: How proxy socket interface look like?} My initial plan is to place prefix "condor_". inet_ntop becomes condor_inet_ntop. connect becomes condor_connect. There are some functions that already has these kind of name such as condor_inet_ntoa() in internet.c. It seems these already existing HTCondor-specific functions are obsolete, either. Thus, we may remove or convert existing condor_ prefixed socket functions. I do not think we need any class that encapsulating those BSD socket interfaces. The reason is that I do not change any semantics of BSD socket interface but only syntax. {section: NO_DNS proxy functions} In condor_netdb.c, there are already proxy functions. {code} struct hostent * condor_gethostbyname(const char *name); struct hostent * condor_gethostbyaddr(const char *addr, SOCKET_LENGTH_TYPE len, int type); int condor_gethostname(char *name, size_t namelen); {endcode} Their original function, gethostbyname, gethostbyaddr and gethostname are already obsoletes. New proxy functions should care about NO_DNS options as well. And also they should not contain obsolete functions. {section: Converting each socket functions } Now, I would like to enumerate all the functions that must be addressed in proxy socket interface. *:inet_aton (standard socket function) It accepts text representation of IPv4 address and then convert to numeric representation. It becomes obsolete and recommended newer function is inet_pton. Calling inet_pton() requires you to choose whether given text representation is IPv4 or IPv6. I think it is required to convert given any IPv4 or IPv6 text string to numerical representation *automatically*. Thus, mechanically changing inet_aton() to inet_pton() is no hope. I would implement a function that automatically detects the address format (IPv4 or IPv6) and then converts it into numerical representation automatically. I would call it condor_inet_pton(). *:is_ipaddr (internet.c) It validates whether given IP address is IP address or not. Its implementation is strictly limited to IPv4. I think it is better to code whole new function rather than converting existing function. *:convert_ip_to_hostname (condor_netdb.c) Note: this is part of the NO_DNS code path and converts 192.168.0.33 to 192-168-0-33.DEFAULT_DOMAIN. This one is little tricky. Given parameter, const char* addr is actually in_addr*. Inside of the function, it explicitly type-cast const char* addr to in_addr*. It calls inet_ntoa and convert every '.' char to '-'. *:getaddrinfo Converting existing code to call getaddrinfo() is not trivial as I expected. This function returns addrinfo structure which embeds sockaddr*. Since it is not sockaddr_in(IPv4 address), the caller must check sockaddr::sin_family to see whether it belongs to IPv4 or IPv6 address. Currently, no HTCondor source codes call getaddrinfo directly. One possible way to deal with getaddrinfo() is that define condor_addrinfo that contains IpAddr class instead of sockaddr struct. So, the proxy function may convert addrinfo to condor_addrinfo. It involves changing sockaddr_in to IpAddr class(which is actually sockaddr_in6). However, I am unsure about this unnecessary memory copies and type changes. This function is replacement of gethostbyname*, getgetservbyname. As noted earlier, there are 165 lines of codes that calling gethostbyname. However, gethostbyname returns struct hostent which does not embed sockaddr but only char* strings. So, these are not really concern. {code} struct hostent *gethostbyname(const char *name); struct hostent { char *h_name; /* Official name of host. */ char **h_aliases; /* Alias list. */ int h_addrtype; /* Host address type. */ int h_length; /* Length of address. */ char **h_addr_list; /* List of addresses from name server. */ #define h_addr h_addr_list[0] /* Address, for backward compatibility. */ }; int getaddrinfo(const char *node, const char *service, const struct addrinfo *hints, struct addrinfo **res); void freeaddrinfo(struct addrinfo *res); struct addrinfo { int ai_flags; int ai_family; int ai_socktype; int ai_protocol; size_t ai_addrlen; struct sockaddr *ai_addr; char *ai_canonname; /* canonical name */ struct addrinfo *ai_next; /* this struct can form a linked list */ }; {endcode} {section: Memo on conversion} Remember that socket() does accept AF_INET and AF_INET6. You should change it appropriately as well. If the user's intention is obvious like following code, I could write a wrapper function that deals automatically. However, I must consider more complicated situations like calling socket() at one site and passing the socket descriptor to elsewhere. And then calls connect() or accept() on it. {code} static SOCKET tcp_connect( const char *host, int port ) { struct hostent *h; struct sockaddr_in address; int success; SOCKET fd; if(!initialize_sockets()) return INVALID_SOCKET; h = gethostbyname(host); if(!h) return INVALID_SOCKET; address.sin_port = htons((unsigned short)port); address.sin_family = h->h_addrtype; memcpy(&address.sin_addr.s_addr,h->h_addr_list[0],sizeof(address.sin_addr.s_addr)); #if defined(WIN32) // Create the socket with no overlapped I/0 so we can later associate the socket // with a valid file descripter using _open_osfhandle. fd = WSASocket(AF_INET, SOCK_STREAM, 0, NULL, 0, 0); #else fd = socket( AF_INET, SOCK_STREAM, 0 ); #endif if(fd == INVALID_SOCKET) return INVALID_SOCKET; success = connect( fd, (struct sockaddr *) &address, sizeof(address) ); if(success == SOCKET_ERROR) { closesocket(fd); return INVALID_SOCKET; } return fd; } {endcode} {section: IPv6 support in External Libraries} {subsection: Unsupported} *:srb : IPv4 only. SRB is used by stork, file transfer software. It is developed by the DICE group. (a combination of U of North Carolina at Chapel Hill and UC San Diego) http://www.sdsc.edu/srb/index.php/Main_Page I guess for now we cannot use SRB on IPv6 port. I hope it won't be any big problem. *:gcb : IPv4 only. {subsection: IPv6 compliant} *:drmaa : it seems it does not call BSD socket functions directly. *:globus : they do use mix of sockaddr_in and sockaddr_in6. in their documentation, they have options to enable ipv6 to use. *:cream : i couldn't find any info about ipv6. So I manually scanned their source code (cream-c-api). There was no sign of using BSD socket interfaces at all. *:blahp : depends on Globus but it does not call socket functions directly. [from Jaime] *:unicoregahp : same argument as blahp, [from Jaime] *:curl : file transfer library over URL format, fully support *:glibc : gnu c runtime library *:krb5 : They said "Partial support for IPv6 addresses has been added." in 2003. Full support right now? *:man : manual page packing *:classads *:libvirt : fully support *:openssl : fully support *:coredumper *:expat : xml parser *:gsoap : according to their homepage, they support IPv6. *:libxml2 : xml library *:pcre : regular expression library *:voms : does not related [from zach] *:hadoop : hadoop is based on Java Runtime and fully supports IPv6. *:linuxlibcheaders : surely, it does not matter *:postgresql : full support *:zlib : compression library {section: Todo} Manageable daily work-list. 1:what is CEDAR socket? get to know this 1:currently, i have no idea how to divide source codes into chunks. so, rather start with condor_util_lib, which deals heavily with IPv4 address. 1::convert is_ipaddr function 1::convert convert_ip_to_hostname function 1:discuss with staffs about new sinful string format. <1.2.3.4:80> will not work! encapsulate ipv6 address by [] ? like <[1234:abcd:1234::5678]:5000> ?? 1:remove every obsolete interface. 1:change every incidents where use sockaddr,sockaddr_in, and int. 1:change IpVerify not to depend on IPv4 address 1:think about test plan since IPv6 port is kind of overhaul. changing occurs in many places. {section: Done} *:Examine external libraries that deals with network. gSOAP, OpenSSL, and what else? *:investigate ipv6 address handling in C++ Boost library *:investigate security and authentication code that deals IPv4 address directly. IP verifier? *:investigate every source code where calls obsolete BSD socket interface. *:make a git branch *:IpAddr class, unified address class *:take a look at unit test {section: Note} *:In a server, we want to accept both IPv4 and IPv6 connections in single program(process). There are two choices with dealing IPv4 and IPv6 co-existence. 1::We may create only one IPv6 socket. Then, OS will fake a server program with IPv6-IPv4 mapped region. ffff::1.2.3.4 is really IPv4 host and came through IPv4 router and everything. 1::We may create two sockets, IPv4 and IPv6. Call non-blocking accept to both of them. And poll() or select() or kqueue() on both sockets. In current Linux implementation, I think we do not need this kind of complexity. Linux kernels deal smoothly with faked IPv6 address. I expect it will be true for commercial Unix like AIX and Solaris. *:internet.h seems to have a lot of IPv6 incompatible functions. *:why do you call as sinful string? what does it mean 'sinful'? - Now that I know. This is HTCondor specific term. We need to extend definition to include IPv6 address. For example, <[a:b:c:d:e:f:g:h]:pppp> would work. *:do we really need unified ip address class? IPv6 address space contains IPv4 and there is 1-to-1 mapping. Current answer is no. But, we may need a 'type' flag. *:Do you have any good arguments for having network-byte-order IP address instead of having host-byte-order IP address? *:check point server and syscall library. will they eventually replaced by DMTCP? they contains IPv4 functions but maybe I don't need to care about them in this project. {section: Timelog} 2010 Jan, 12 : dealt with proxy functions. mostly internet.c and condor_netdb.c. wrote simple IPv6 test case for unit test FTEST_is_ip_addr. Jan, 18 : attempting to convert function by function. mostly targeting utility functions first. ... (omissioned) ... Jan, 25 : tackling with inet_ntoa Jan, 26 : completed condor_inet_pton and wrote unit test. Jan, 28 : tackling with internet.c, mostly is_ip_addr related.