Page History
- 2014-Nov-19 14:33 johnkn
- 2012-Nov-13 15:13 adesmet
- 2011-Sep-28 11:51 adesmet
- 2011-Sep-28 11:04 adesmet
- 2011-Sep-28 10:26 adesmet
- 2011-Sep-28 10:25 adesmet
- 2011-Aug-30 16:55 adesmet
- 2011-Aug-30 16:53 adesmet
- 2011-Aug-29 17:46 adesmet
- 2011-Aug-29 17:41 adesmet
- 2011-Aug-29 17:36 adesmet
- 2011-Aug-29 12:40 adesmet
- 2011-Aug-29 12:40 adesmet
- 2011-Aug-29 12:38 adesmet
- 2011-Aug-29 11:40 adesmet
- 2011-Aug-29 11:36 adesmet
- 2009-Apr-02 10:40 jfrey
- 2009-Feb-27 14:34 jfrey
- 2009-Feb-27 14:23 jfrey
- 2009-Feb-18 14:14 psilord
- 2009-Jan-16 14:52 michaelb
Adding a new external package to condor
This document describes how to add a new external package to the Condor build. There are a number of elements to this task which are described below:
- Packaging the external so it can be built by the build system
- Setting up the source tarball
- Making the build script
- Build script environment
- Build script syntax
- Build script exit codes
- Build script example
- Handling patches (optional)
- Telling the build system about the new package
- Changing autoconf-related stuff
- Changing imake-related stuff
- Telling CVS about the new external
- Checking it into the externals tree (to the trunk)
- Checking in Condor build system changes (to a specific branch)
- Adding it to the appropriate CVS modules
- Pre-Building the new external in AFS(optional)
Before getting into the specifics, here's the 10,000 foot view of how the Condor build system deals with externals:
- autoconf-generated
configure
script decides what versions of what externals we're trying to use for a given version of Condor configure
script puts its results inconfig/externals.cf
andconfig/config.sh
imake
usesconfig/externals.cf
to define make variables that point to the necessary parts of each external (usually the installed output of the external build)src/Imakefile
contains rules that specify what externals actually need to be built (and optionally, what other externals they depend on). These make rules are fired off bymake externals
, which is a rule that the top-levelall
make rule depends on, early in the build.- Each external package that's being built is managed via
externals/build_external
which is the main script that handles common tasks for all externals. This script is responsible for setting up some directories, unpacking source, maintaining a build log, and a few other things. - The
externals/build_external
script invokes a build script that is specific to a given version of a given external. This build script handles the details of building a specific external and installing the libraries, header files, and/or anything else we need from the package into a location where the Condor build can use it. - If the build script is successful,
externals/build_external
touches a file that says the external was happily built and installed. The Condor build Makefiles can depend on this "trigger" file to ensure that anything that needs the external is built after the external was successfully built.
Understanding the above overview will help make sense of the specific steps needed for modifying the build system to know about and use a new external.
In general, the process of adding a new version of an existing external is very similar to if you're adding a whole new external. However, in any point in this instructions where something extra is required, that will be made clear...
Naming schemes and conventions
Before we get into the nuts and bolts, there are a few naming conventions that need to be defined and clarified.
Each external package is identified by 2 things:
- Name (e.g. "globus" or "krb5")
- Version (e.g. "1.2.7")
These two parts of the package identification are important in
numerous places, and it's important to keep it all straight. For the
rest of this document anywhere you see: [name]
or
[version]
, you should substitute the right thing based on the
package you're adding.
Regarding the name: it should be descriptive, but not too long/verbose. "z" is not good. "zlib-compression-library" isn't either. usually it's pretty obvious. if you have any questions ASK someone else BEFORE you start adding, since it'll eventually require adding a new directory to our repository, and that's impossible to undo...
Regarding the version: ideally, this should be the official version number of the external package as decided by its authors. If we patch a given external, that doesn't change the official version we're basing the package on, and that's what the version number should be.
However, if you want to add new patches to an existing external, you should change the version number and add a new external! We do NOT want to have multiple things which are different code to be using the same version!
Again, if you have any questions or are uncertain, just ask.
Finally, the combination of [name]-[version]
is often
referred to in the build system (and this document) as the package
name, and you'll see it listed as [package-name]
Packaging the external so it can be built by the build system
Each external package used to build Condor lives in a unique
directory as part of the externals
directory tree. The basic
directory layout of externals
is as follows:
externals/ /build /install /triggers /bundles/ /[name] /[version]
The build
directory is a temporary build sandbox, which
we'll talk about more later. The install
directory is where
each external package installs its build output so it can be used by
the Condor build. The triggers
directory holds the trigger
files (mentioned in the overview) that determine if a given external
was successfully built. Want to have your Condor workspace rebuild the
zlib external? Just remove the
externals/triggers/zlib-[version]
file and the
src/Imakefile
will ensure that a top-level "make" will
rebuild zlib...
The bundles
directory contains subdirectories for each
kind of package, and each package directory has subdirectories for
each version of that package. For example, at the time of this
writing, we've got 5 different versions of the glibc external, each
living in their own subdirectory of externals/bundles/glibc
.
externals/ /bundles/ /glibc/ /2.1.1 /2.2.2 /2.2.93 /2.3.2 /2.3.2.27.9.7
Inside each version-specific subdirectory, are 2 main things:
- Original source tarball (ideally, exactly what you'd download from the authors distribution, unmodified)
- Build script
Optionally, there may be patch file(s) which include modifications we had to make to the package to get it to work with Condor.
Each of these things are addressed in the following sections...
Setting up the source tarball
Ideally, there's nothing you should need to do to modify the source
tarball of the external package. We want the original, unmodified
source whenever possible. However, the name of the tarball is
important, since the Condor build system itself makes assumptions
about the name so that the build_externals
script can untar
the tarball for you (one less thing for your build-script to worry
about for yourself). So, the source tarball must be
named "[name]-[version].tar.gz
" (and, needless to say, it must
be a real gzip'ed tar file). For example, "krb5-1.2.7.tar.gz
".
Making the build script
When the Condor build is trying to build your external, first it
will create a temporary sandbox build directory. The
[name]-[version].tar.gz will be untarred into the sandbox. Then, the
build_[name]-[version]
script will be invoked with the
current working directory set to the sandbox build directory. This
script is responsible for building the external package in whatever
way makes sense. The calling layer assumes the responsibility of
cleaning up the build sandbox. It will also keep track of whether or
not things need to be rebuilt.
Build script environment variables
In addition to being born with the build sandbox directory as the current working directory, the following environment variables will be set (these are literal... no substitution of package name or anything):
$PACKAGE_NAME # the name of this package $PACKAGE_DEBUG # empty if release, '-g' if debug build $PACKAGE_BUILD_DIR # the sandbox build directory $PACKAGE_INSTALL_DIR # where to put the results $EXTERNALS_INSTALL_DIR # full path to root of externals/install $EXTERNALS_CONFIG # full path to config.sh with config variables
$PACKAGE_NAME
is the [name]-[version]
identifying
string for your package.
$PACKAGE_BUILD_DIR
is a subdirectory of
externals/build
, named with the package-name
.
This is just a temporary sandbox directory, and
build_external
will remove this directory and all its
contents as soon as your build script exits.
$PACKAGE_INSTALL_DIR
is a subdirectory of
externals/install
, named with the package-name
as
well. This directory won't necessarily exist when your build script
is spawned, so your script is responsible for creating this directory
in whatever way is appropriate for your package. Most packages don't
have to worry about it, since make install
takes care of it
for you. Some packages need to copy a whole tree to this location, so
we don't want to make you rmdir
if you don't want it there in
the first place. However, the parent of this directory (the directory
path stored in $EXTERNALS_INSTALL_DIR
) is guaranteed to
exist, so you don't need to worry about mkdir -p
.
If your build script wants to know things that the Condor build
system determines at run-time (for example, what globus flavor we're
trying to build), it can source the file specified in
$EXTERNALS_CONFIG
(details on sourcing this file below).
This is a simple bourne shell syntax file that defines a bunch of
shell variables that hold information that the configure
script discovers. If your external needs to know something that the
configure
script figures out but that isn't yet in
config/config.sh
, all you have to do is add a new line to
config/config.sh.in
to define the variable you need. Also,
note that if you want to pass any of these variables onto the
configure
script or build system of your package, you either
have to pass them in as arguments or manually export them yourself as
environment varialbes (by default, you just get shell variables in
your script, not environment variables).
Build script syntax
YOUR BUILD SCRIPT MUST BE PLAIN OLD BOURNE SHELL!!!.
Do NOT use any bash syntax, since it won't work on all 15+
Condor platforms. For example, source
is not portable across
all versions of bourne shell. The correct way to include this file in
your build script is to use:
. $EXTERNALS_CONFIG
Similarly, to set an environment variable, you can NOT do this:
export FOO=bar
You MUST use:
FOO=bar export FOO
Another shell idiom that does NOT work is using this:
if( ! make ); then blah blah; fi
You MUST use:
make if [ $? -ne 0 ] then blah blah fi
Build script exit codes
If your script encounters an error during the build, exit with status 1.
After building, you should run any tests that are available.
If any tests do not pass or any other error occurs, exit with status 2.
After passing tests, you script should install the package into the
directory $PACKAGE_INSTALL_DIR
in whatever format is
customary for that package.
If there is a problem during installation, exit with status 3.
If there is any problem specific to your package, exit with status 10 or higher. Status 4-9 is reserved for future use. You can only exit with status 0 if everything is fully installed and working (to the best of your knowledge).
Build script example
Here is an example build_generic_package
, which would
build most things that follow the common tar and 'configure / make /
make test / make install' convention:
#!/bin/sh ############# build_generic_package cd $PACKAGE_NAME/src ./configure --prefix=$PACKAGE_INSTALL_DIR --with-ccopts=$PACKAGE_DEBUG make if [ $? -ne 0 ] then echo "make failed" exit 1 fi make test if [ $? -ne 0 ] then echo "test failed" exit 2 fi make install if [ $? -ne 0 ] then echo "install failed" exit 3 fi exit 0 ############# end of build_generic_package
Handling patches (optional)
If your external needs to be patched, the preferred method for
dealing with it is to create patch files, and put those patches into
the version-specific subdirectory. Then, the build-script for the
package can invoke patch
as needed. An example to look at
would be the krb5-1.2.5
external
(externals/bundles/krb5/1.2.5/
)
Again, if you want to add additional patches to an existing
external, you MUST make an entirely new external package with a
different version number (e.g. something like krb5-1.2.5.pl1
)
so that we can tell the difference between the two versions. This is
a little wasteful of space, unfortunately, but there's no way around
that at this time.
Telling the build system about the new package
Once your package is setup in the externals tree and the build script ready, you've got to tell the Condor build system about the new package. There are a few separate places that this needs to happen.
Changing autoconf-related stuff
The first step in integrating a new external with the build system
is to modify src/configure.ac
, the input file for
autoconf
. Near the bottom of this file, there's a section of
tests that determine what version of each external we want to use.
The section begins with the following code block:
############################################################ # What versions of what externals do we need ############################################################ AC_MSG_NOTICE([determining versions of external packages])
Some of the "tests" are trivial, they just define a version to use
directly. Other tests do something more fancy, like choose a version
depending on the platform we're on. Ultimately, autoconf
is
just a fancy way of using M4 macros to make a giant bourne shell
script. If you following the existing syntax and examples already in
this section, you should be in good shape. If you have trouble,
either check out the autoconf
documentation (for example autoconf 2.57).
The important thing is that once you know what version of the external you need, that you call our special autoconf macro,
CHECK_EXTERNAL()
, which handles everything else for
you.
For example, here's a simple case, zlib:
CHECK_EXTERNAL([zlib],[1.1.3], [soft])
This handles the following tasks for you:
- Defines what version of zlib we need.
- Checks that the given version of zlib exists in the externals tree the build is configured to use.
- Prints out a message to the screen about the version of zlib we're going to be using.
- Tells autoconf that any file it is producing should
substitute any occurrence of
@ext_zlib_version@
with the value "zlib-1.1.3
". - Provides you with
HAVE_EXT_ZLIB
for use in =#ifdef='s, if the zlib is found. - Tells autoconf that the external is a soft requirement. More on this later.
If you're just changing the version of an existing external, that's
probably all you'll have to do to the autoconf
stuff, and you
can skip right to the discussion of CVS
changes.
However, if you're adding a whole new external package, there are a
few more steps (both for autoconf
and imake
, so read
on...
To add a whole new kind of external, you've got to understand a
bit about that [soft]
argument (the requirement level), modify
config/externals.cf.in
and config/config.sh.in
to
hold the results of the new test you added in
src/configure.ac
. Again, if you just edit the files and
follow the existing examples, you'll probably be in fine shape. But,
for the sake of complete docs, I'll explain in a little more detail
here:
requirement levels
There are three requirement levels, and they determine how
dependent Condor is on an external. The levels are soft
,
hard
, and optional
. The soft
level is the
most common and is likely what you need to specify. It means that
Condor can compile and operate just fine without the external. When
configure
is run it will look for soft
requirements,
but will only give a warning if they are not available. This is in
contrast to hard
requirements, which are always
required. Condor either will not compile or run without them. They
MUST be present. Not being present will cause configure
to
fail. You want to avoid adding hard
externals to Condor. The
third level, optional
, is just as uncommon as hard
,
if not more. It operates almost exactly as soft
with one
exception: there is a --with-soft-is-hard
option that can be
given to configure
to treat all soft
requirements as
if they were hard
requirements. This option does not change
how optional
externals are treated. So, you really want your
external to be a soft
, unless you have a really good reason
for it not to be.
config/externals.cf.in
The most important thing is to define a new make
variable
to hold the version of the external you added. For example, here's
what happens for zlib:
EXT_ZLIB_VERSION = @ext_zlib_version@
The rest of the build system that needs to know how to find zlib can now use this variable. Almost always, what we really care about is the installed output of the external build. This is a common way to deal with that:
ZLIB_DIR = $(EXT_INSTALL)/$(EXT_ZLIB_VERSION) ZLIB_LIB = $(ZLIB_DIR)/lib ZLIB_INC = $(ZLIB_DIR)/include
Obviously, the details depend on the specifics of a given package, but
in general, something like the above is what you want to do. This
way, we have the full path to the lib
and include
directories for the external, and each of those make
variables can be used in regular Condor imake
input files so
that the Condor build can use the external.
config/config.sh.in
The config/config.sh.in
file is converted by
configure
into config/config.sh
. The full path to
this config.sh
file is passed into the
externals/build_external
script by the Condor build system,
which in turn sets it as the $EXTERNALS_CONFIG
environment
variable. This way, the package-specific build scripts can use
information that configure
discovers if that information is
necessary to build an external. In general, this is only needed when
a given external depends on a different external. For example, the
gahp external needs to know where the globus external was
installed, and what "globus flavor" was built. The gahp external also
needs to know if the platform we're on is trying to build a
statically-linked version of Condor or not. So,
config/config.sh
defines the following variables:
... GLOBUS_FLAVOR=... EXT_GLOBUS_VERSION=... HAS_STATIC=... ...
Inside build_gahp-1.0.10
, we use the following code:
. $EXTERNALS_CONFIG FLAVOR="--with-flavor=$GLOBUS_FLAVOR" GLOBUS="--with-globus=$EXTERNALS_INSTALL_DIR/$EXT_GLOBUS_VERSION" STATIC_BUILD="--with-static=$HAS_STATIC" cd $PACKAGE_BUILD_DIR/$PACKAGE_NAME/ ./configure --prefix=$PACKAGE_INSTALL_DIR $GLOBUS $FLAVOR $STATIC_BUILD
Changing imake-related stuff
NOTE: if you're just changing the version of an existing external,
you probably don't have to mess with any imake
stuff at all.
Once autoconf
knows all about your new external, and the
various make
variables have been set in the files in
config
, there are a few minor changes to the imake
input files themselves that need to be made so the Condor build knows
about the external. In particular, the src/Imakefile
needs
to know about your external.
To do this, all you have to do is add a new ext_target
to
the file. This magic imake
rule handles all the details of
ensuring the Condor build depends on your external, and that
externals/build_external
is invoked to build your external at
the appropriate time. This target takes two arguments, the package
name (which should be held in the make
variable
EXT_[NAME]_VERSION
), and any externals your package depends
on.
Here's the boring zlib example. There, nothing to it, since we always want this external built, and it doesn't depend on anything else:
ext_target(EXT_ZLIB_VERSION,$(NULL))
Here's the complicated globus example. It depends on the GPT
external, and we only want to build it if the HAVE_EXT_GLOBUS
imake #define is set, which happens when configure
finds the
globus external:
#if HAVE_EXT_GLOBUS ext_target(EXT_GLOBUS_VERSION,$(EXT_TRIGGER)/$(EXT_GPT_VERSION)) ... #endif
Note the use of $(EXT_TRIGGER)
. That's how imake
(and therefore, make
) know a given external was successfully
built. $(EXT_TRIGGER)
holds the full path to the
triggers
directory (described above). Each trigger file is
just named with the package name. So, by making the globus external
depend on $(EXT_TRIGGER)/$(EXT_GPT_VERSION)
, we ensure that
GPT is built before Globus.
If your external depends on multiple other externals, all you have
to do is include multiple trigger files (separated by spaces) in the
second argument to ext_target
. For example:
ext_target(EXT_FOO_VERSION,$(EXT_TRIGGER)/$(EXT_BAR_VERSION) $(EXT_TRIGGER)/$(EXT_BAZ_VERSION))
Finally, there's the tricky subject of exactly how the Condor
imake
system should use your external and integrate it with
the rest of the Condor source. Eventually we'll have a better answer
for that question here. For now, if it's not totally obvious, just
talk to someone else on the staff and try to come up with something
good. When in doubt, ask Derek.
Once all this is done, you're in a position to fully test your new
external with a Condor build. You'll have to re-run autoconf
to generate src/configure
and re-run the
src/configure
script itself. After that, a top-level
make
inside src
should be all you need to see your
external package built. Once your external build is working and the
Condor source sees and uses the new external, you're ready to commit
your changes to CVS...
Telling CVS about the new external
This is mostly obvious, boring stuff, and I assume you know how to use CVS. I'm just including this section so that you don't forget any of these final steps...
Checking it into the externals tree
The externals
tree lives on the trunk of the Condor CVS
repository. It is never branched, merged, etc. So, all externals are
in theory visible from all Condor CVS branches. When you add your new
directory into externals/bundles/[name]/[version]
you should
ensure that you're committing your new files to the trunk. You should
add the directory, then do a cvs add
to the source tarball
(cvs already knows files that end in .gz are binary, so you don't have
to worry about that), the build script, and any patches you've made.
Once that's done, you can cvs commit
as normal.
Checking in Condor build system changes
The changes you made to src/configure.ac
,
src/Imakefile
(if any), and any changes to files in
config
must be committed to a specific Condor CVS branch.
Fundamentally, it's the fact that
src/configure.ac
is branched and merged with the rest of the
Condor source that enables us to know exactly what versions of each
external were used for a given version of Condor. So, when you're
committing all those changes to the build system (and to the rest of
the Condor source to take advantage of the new external), you must do
so to a real branch. In fact, most of the time, you'll want to
create a new branch off the main development branch at the time to
deal with adding your new external. That way, we can test building
your new external and all related changes on all our platforms,
without breaking the build on the main release branch at the time.
Adding it to the appropriate CVS modules
Finally (and this applies to both a new version of an existing external and adding a whole new kind of external), you should add your new external to the appropriate CVS module(s). Even though your new external lives on the trunk and is therefore visible by all Condor branches, it doesn't mean we actually want to see your external everywhere.
To solve this issue, we rely on a number of CVS modules to select
the versions of the externals we care about on each main Condor CVS
branch. For example, V6_6_EXT
holds all the externals we
need for building the V6_6-branch
of Condor. So, if you
added a new external to the V6_7-branch
, you'd want to add
another line to the V6_7_EXT
CVS module.
To modify a CVS module, all you have to do is this:
% cd /tmp % cvs co CVSROOT/modules % <edit> CVSROOT/modules % cvs commit CVSROOT/modules </edit>
As always, if you open up the file and look, the syntax should be pretty obvious. The main thing is that you remember to do this step at all, and that you add your external to the right module(s).
Things might get a little tricky if you're replacing an old version
of a given external with a new one. In that case, be sure you don't
break any of the version-specific historical modules
(e.g. V6_6_2_EXT
) when you want to remove the old version.
In this case, V6_6_2_EXT
is defined relative to
V6_6_COMMON_EXT
. So, if you want to remove
externals/bundles/globus/2.2.4
from the V6_6_3_EXT module,
you'll probably have to remove globus/2.2.4
from
V6_6_COMMON_EXT
and manually add it back to all the
version-specific modules that used to include it. An example of this
is that we changed the version of the gahp external between
V6_6_0
and V6_6_1
, so we had to remove the gahp from
V6_6_COMMON_EXT
since the same version of it was no longer
common to all the 6.6.x modules.
Pre-Building the new external in AFS (optional)
As a final (optional) step, you should probably make sure the
pre-built externals tree in AFS
(/p/condor/workspaces/externals
) is up to date and that the
new external has been pre-built on all the platforms we care about.
All you've got to do is:
- Update
/p/condor/workspaces/externals
- Start a build on each platform we care about
To update the externals tree, just do this:
% cd /p/condor/workspaces % cvs co externals/bundles
Once that's been updated, all you have to do is start a build of
the Condor source (from the branch where you checked in your build
system changes for the new external) on each kind of machine we care
about. Basically, any platform with an @sys
directory in
/p/condor/workspaces/externals/sys
is what you'd need to
worry about.
Just make sure the build on each platform uses
/p/condor/workspaces/externals
for the externals. If you do
not check out a local copy of externals into either your source or
build workspaces, our configure
script will use the tree in
AFS by default. Otherwise, you can always use this:
% ./configure --with-externals=/p/condor/workspaces/externals
Once you start the build, the externals will be built first. Assuming everyone else is following these directions, the only external that will need to be built is the one you just added. This will ensure that all the developers using this pre-built tree won't have any problems as a result of your new external.