HTCondorWiki: Build Modernization

Page History

Introduction

This wiki page contains information regarding build modernization and consolidation. For details on progress see Ticket #779

Goals and Objectives

The goal of updating the condor build mechanism is far more reaching then using the latest wiz-bang tool to build code, and is meant to address the following issues:

Build consistency across platforms (Including Windows): Currently Imake does a good job for build consistency across *nix variants, however there is an entirely different set of mechanics for dealing with builds on Windows, which few people really understand. To make matters a bit more confusing, is that the mechanism by which we build & test on Windows is quite non-standard which presents an issue as condor becomes a more open source project. This plurality causes several issues in maintaining two desperate build files, with completely different mechanisms. It would be in the best interests for the group in the long run to employ a single build mechanism which has support on all condor platforms.
Support for unit testing within the build: For the purposes of testing it is advantageous to have a build tool which supports running automated unit tests as part of a "unit" build, with a minimal amount of external overhead.
Packaging: Many build systems provide automated packaging for it's native environment. This can be highly useful as it eliminates and automates custom build steps and scripts and makes it easier to maintain. Having this requirement can reduce the overhead on condor developers and allow them to focus on bug fixes and new features. It also allows flight worthy to get out of the packaging business and simplifies the consumption for downstream folks.

Possible Options

If we use the afore mentioned goals as hard requirements, then it eliminates a lot of possible choices, and only a couple remain:

SCons: SCons is a fairly powerful tool build atop of a python, which allows you to pretty much do anything. The only major disadvantage is the issue of packaging, and size of adoption within the open source community. There are also issues when dealing with exiting auto-tools style builds.
bjam: Created as a replacement/extension to perforce JAM it has it's own syntax and supports numerous platforms, it also does not provide flexible packaging features, and lacks the transition glue to move from an auto-tools style build.
CMake - winner: CMake is widely adopted in the open source community as a x-platform replacement for auto-tools style builds. It has it's own Macro-style syntax, but unlike the afore mentioned tools it is a generator similar to Imake and it works with existing tools (make, visual studio). Another powerful tool in the "CMake warchest" is CPack, which allows for various methods of packaging, across multiple OS distributions.

Getting Started

1.) You will need to obtain CMake 2.6 or > for the target platform you are working on.

2.) Checkout origin/CMAKE, for windows you will need git 1.6.4 or greater.

2.5) Windows only: Because the temporary structure is softlinks to the original files, this causes an issue on windows. So for the time being you will need to flatten the linking using cygwin.

cp -r -L new_build new_build_win (I'm debating on eliminating the softlinks)

3.) Because the build expects that you have certain pre-reqs installed and defaults to a clipped build (atm). You will likely need to install the default set of clipped && proper libraries. On windows this means staging the externals into a location until I can get around to writing a pre-configure script.

4.) Navigate to CONDOR_SRC/src/new_build

5.) cmake ./CMakeLists.txt

6.) make || open the visual studio project & build If you wish to see the native output type make VERBOSE=1

Contributing

All of the updated cmake files exist under src/new_build and are open to contribution. The current branch tracks against master(7.5 series) and all files under new_build are a series of softlinks against the originals to allow for easy merge-outs, and to allow for parallel builds in case it is needed. There are a substantial number of changes and cruft removal in this branch so there may need to be a bit of work to get the old build to function in this new branch, as I have not made it a priority atm.

Conventions

#include <lib>/header.h (no more relative includes)
Compiles fully in C++
Proper use of extern
Minimal linkage, explicitly specify
Platform specific files are suffixed as to allow glob macros to add to target (.windows, .unix)
condor specific macros are located under build/cmake

Packaging

CMake's compliment, CPack, supports a large number of packaging types which can be found here, and it is currently in its infancy in the code base. There are two targets which are created when you include(CPack):

make package
make package_source

The road ahead is to create a stub'd out CPackOption.cmake.in file and use cmakes configure options to fill in the gaps based on the target platform and environment, thereby reducing the cruft in the master CMakeLists.txt and providing the wiz-bang feature of platform specific package generation.

TODO

setup so all gen files & compilation files go to a single location for easy removal (similar to bjam) single checkout, w/multiple build products.
Setup packaging configuration.
Stage Externals as part of a configure.
Deal with softlinks on windows
Clean up remnants of gsoap
add checks for perl, awk, and flex (right now I believe it is assumed)

Links

CMake Official Documentation

Google Code Search for open source examples