Adding New External Package

Not logged in

• Browse
• Help
• Home
• Login
• Reports
• Search
• Timeline
• Wiki

• Contents
• No-Diff
• Text

 *: Packaging the external so it can be built by the build system
 *:: Setting up the source tarball
 *:: Putting the source tarball on the web server
-*:: Setting up the URLS file
 *:: Making the build script
 *::: Build script environment
 *::: Build script syntax
@@ -26,24 +25,9 @@
 
 *:The actual source tarball is hosted on a public web server.
 *:cmake decides which externals we need. The version is part of the cmake information for the branch.
-*:(?? May be outdated ??) Each external package that's being built is managed via
-    =externals/build_external= which is the main script that
-    handles common tasks for all externals.  This script is
-    responsible for setting up some directories, downloading and
-    unpacking the source tarball,
-    maintaining a build log, and a few other things.
-*:(?? May be outdated ??) The =externals/build_external= script invokes a
-    build script that is specific to a given version of a given
-    external.  This build script handles the details of building a
-    specific external and installing the libraries, header files,
-    and/or anything else we need from the package into a location
-    where the Condor build can use it.
-*:(?? May be outdated ??)If the build script is successful,
-    =externals/build_external= touches a file that says the
-    external was happily built and installed.  The Condor build
-    Makefiles can depend on this "trigger" file to ensure that
-    anything that needs the external is built after the external was
-    successfully built.
+*: /build/cmake/CondorConfigure.cmake looks for the external-specific CMakeLists.txt
+*: The individual external CMakeLists.txt describe how to download, patch, and build the external.
+*: Externals are built and stored in /scratch/condor-externals; you can change this default location with something like -DEXTERNAL_STAGE:PATH=/path/to/a/private/directory when you invoke cmake.
 
 Understanding the above overview will help make sense of the
 specific steps needed for modifying the build system to know about and
@@ -51,8 +35,6 @@
 
 In general, the process of adding a new version of an existing
 external is very similar to if you're adding a whole new external.
-However, in any point in this instructions where something extra is
-required, that will be made clear...
 
 
 ----
@@ -85,9 +67,15 @@
 *However, if you want to add new patches to an existing external,
 you should change the version number and add a new external!  We do
 NOT want to have multiple things which are different code to be using
-the same version!*
+the same version!*  If differt versions share the same name,
+externals caching may lead to builds using the wrong patches.
 
-When adding new patches to an existing external, add a =-p1= to the =[version]= to get your new version. If the =[version]= already ends in =-p<number>=, increment the number for your new version. You should continue to use the original tarball (with it's original name).
+When adding new patches to an existing external, add a =-p1= to the =[version]= to get your new version. If the =[version]= already ends in =-p<number>=, increment the number for your new version.
+
+You should continue to use the original tarball (with it's original
+name).  *At the moment this is not possible*, as the CMakeLists for most
+(all?) of the externals assume that the tarball name, the directory
+name inside of the tarball, and the version are identical.
 
 If you have any questions or are uncertain, just ask.  Good people to ask are *TODO*.
 
@@ -114,7 +102,9 @@
 
 The =bundles= directory contains subdirectories for each
 kind of package, and each package directory has a subdirectory for
-the version of that package.  Some packages may have multiple versions *TODO Why?*  For example, at the time of this
+the version of that package.  Some packages may have multiple versions
+because different operating systems require different versions.
+For example, at the time of this
 writing, we've got 5 different versions of the glibc external, each
 living in their own subdirectory of =externals/bundles/glibc=.
 
@@ -132,7 +122,8 @@
 Inside each version-specific subdirectory, are several things:
 
 *:=CMakeLists.txt= which describes how to build the external. It will include the URL to download the source tarball(s) from. (ideally, exactly what you'd download from the authors distribution, unmodified)
-*:Build script
+*:=build_[package-name]= A build script, usually partnered with
+a Windows specific script in the form =build_[package-name].bat=.
 
 Optionally, there may be patch file(s) which include modifications
 we had to make to the package to get it to work with Condor.
@@ -141,33 +132,34 @@
 
 {subsection: Setting up the source tarball}
 
-Ideally, there's nothing you should need to do to modify the source
+In theory there's nothing you should need to do to modify the source
 tarball of the external package.  We want the original, unmodified
-source whenever possible.  However, the name of the tarball is
-important, since it's easier to update the build script with the new filename. So, the source tarball should be
-named "=[name]-[version].tar.gz=" (and, needless to say, it must
-be a real gzip'ed tar file).  For example, "=krb5-1.2.7.tar.gz=".
+source whenever possible.
 
-*TODO: May be out of date, the -p0 may be mandatory now* An important exception is that the =-p<number>= at the end of the version
-is optional in the tarball name. For example, the tarball for external =krb5-1.2.7-p1= can be named =krb5-1.2.7.tar.gz= (the same tarball used for external =krb5-1.2.7=).
+In practice you'll need to repackage the tar.
 
-*TODO: adesmet's edits end here*
+First, the name of the tarball must be in the form
+=[package-name].tar.gz=.
+Second, the contents of the tarball must be in a subdirected named
+=[package-name]=.  The current CMakeLists.txt assume this.  For
+example, "=krb5-1.2.8.tar.gz=".
 
 {subsection: Putting the source tarball on the web server}
 
 The source tarballs live on the web server parrot.cs.wisc.edu. They are synced periodically from the following directories on AFS at UW-CS:
 
-{code}
-/p/condor/repository/externals
-/p/condor/repository/externals-private
-{endcode}
+*: =/p/condor/repository/externals=
+*: =/p/condor/repository/externals-private=
+
+*TODO:* How often is this synchronized? Is there a way to force a sync
+faster?  Will it overwrite earlier, wrong versions?
+
+*TODO: adesmet's edits end here*
 
 The latter is for files that can't be publicly distributed. Currently, the only thing there is a LIGO application for testing standard universe. Once synced, the files can be fetched from the following URLS:
 
-{code}
-http://parrot.cs.wisc.edu/externals
-http://parrot.cs.wisc.edu/externals-private
-{endcode}
+*: ={link: http://parrot.cs.wisc.edu/externals http://parrot.cs.wisc.edu/externals }=
+*: ={link: http://parrot.cs.wisc.edu/externals-private http://parrot.cs.wisc.edu/externals-private }=
 
 Once you place a tarball in =/p/condor/repository/externals= or =/p/condor/repository/externals-private=, you shouldn't change it. You must not change it once a mention of it is pushed to the shared git repository. If the contents of your source tarball(s) may change while you're preparing to push your changes, you have two options for temporarily locating the tarballs:
 

This work is supported by NSF under Cooperative Agreement OAC-2030508 as part of the PATh Project. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF. Site built using CVSTrac.