Page History

Turn Off History

This is a living document that enumerates all the steps necessary to release a new version of Condor.

In parallel with this effort the wrangler should prepare the version history

There are a few phases of the actual release:

  1. Preliminary work to freeze the code, make a release branch, etc. (this can take a few days, so start early)
  2. Upgrading the pool(s) and watching for bugs
  3. Release the binaries to the public and announce the release

Building the right binaries

  1. Coordinate a development code-freeze. This involves chasing down people who are trying to finish bug fixes, new features (if it's a development release), porting, or any other core development that is slated for a given release.

  2. Ensure that the parent branch (e.g. V7_0-branch) is building properly on all platforms. If any platforms aren't building, you need to either solve it yourself or track down the person who broke the builds and get them to fix it immediately. This can take days, so start early!

  3. Notify the appropriate people about the update.
    • For a stable release we update the CS pool. Send email to the uwcs-condor@cs.wisc.edu list announcing our plans for the pool upgrade. We like to give users at least a few days notice so that if there are important paper deadlines, etc, they have time to finish their jobs, or to let us know they need us to delay the upgrade.
    • For a developer release we update the NMI pool and CHTC pool. Let the admins of these pools know about the pending release (currently the Condor infrastructure team for NMI, and infrastructure or Dan Bradley for CHTC)

  4. Make a release branch. The main thing to know is that the branch should have the name VX_Y_Z-branch. Here's an example of the steps for a 7.0.3 release, from a known good NMI tag (assuming your working directory is inside of a clone of CONDOR_SRC.git):
    • The preferred method for making a release branch is from a known good build tag on NMI. 
      git branch V7_0_3-branch BUILD-V7_0-branch-2008-6-24
git push origin V7_0_3-branch
    • or, if you'd like to make the branch off of the V7_0-branch: 
      git branch V7_0_3-branch origin/V7_0-branch
git push origin V7_0_3-branch
    • either way, be sure to update GitBranchDescriptions and SourceCodeBranches to document the new branch

Skip the next step, because we need PRE_RELEASE on, in order for NMI to make builds with NMI buildids in it, which NMI needs to install in NMI.

  1. Fix the version string in the release branch. However, in this case, the goal is to remove the "PRE-RELEASE" and other junk from the version string so you're left with an official version string for the final release.
    • git checkout V7_0_3-branch
    • Edit the top-level CMakeLists.txt
      • Set PRE_RELEASE to OFF
    • git diff (to verify the change before you commit it)...
    • git commit -a -m "set official version string for 7.0.3"
    • git push origin V7_0_3-branch

  2. Update CMakeLists.txt on the main-line development branch (e.g. V7_0-branch) to go to the next version number and to include the string "PRE-RELEASE-UWCS".
    • git checkout V7_0-branch
    • Edit the top-level CMakeLists.txt
      • Set PRE_RELEASE to "PRE-RELEASE-UWCS"
    • git diff (to verify the change before you commit it)...
    • Edit the top-level CMakeLists.txt
      • Set VERSION to the new version
    • Edit build/cmake/CondorPackageConfig.cmake
      • change the GUID value for CPACK_WIX_PRODUCT_GUID (at about line 190)
    • git commit -a -m "updated version string for 7.0.4"
    • git push origin V7_0-branch

  3. On the main branch of the manual (not the newly created release branch, but for example, the regular V7_0-branch), update condor-macros.tex to up the version number to the next version, and push those changes to CONDOR_SRC.git.

  4. We’re modifying our process to start with an upgrade request ticket whose body includes specific issues to note for the release. Essentially, the idea is to send in the NMI ticket same new release announcement we are going to send to our user community. This way, the NMI admins act as a 'beta site' for the relevancy of our release announcement. If we have issues upgrading, we’ll update the ticket text so that that it become the email that is ultimately sent to users.

  5. Make sure the nightly build tags are being made on the release branch:
    • cd /p/condor/home/tools/cvs-tools
    • Edit create_build_tag-input
    • Uncomment and modify a section that defines a nightly build tag on the appropriate release branch. See the other examples in the file for details. There's also a very complete comment at the top of the create_build_tag script itself.
    • cvs diff create_build_tag-input (to verify the change before you commit it)...
    • cvs commit -m "added nightly build tags for V6_X_Y-branch" create_build_tag-input

  6. Watch the nightly builds the next morning to ensure everything is working fine. Common problems include:
    • Confusion with the proper branch to pull the manual from for the nightly build tag.

  7. If you'd like to fire off a build immediately, then read the Building Condor with NMI page. You'll need to start the build as cndrauto so follow the directions.

Upgrade the Pools

  1. Settle on a build ID candidate that successfully built on all platforms. Ensure that all tests passed.

  2. Perform the ManualRegressionTests, possibly assigning some to various staff members.

  3. On nmi-s006, pin the build for 365 days using /nmi/bin/nmi_pin: 
    $ nmi_pin --days=365 --force <runid>
    This buildid will be unpinned by the next wrangler when the next revision of Condor comes out in this series. Example: If you are the current release wrangler and have just pinned 7.0.0. The release wrangler for 7.0.1 will unpin the 7.0.0 release.

  4. On nmi-s006, cd to the rundir and remove the .htaccess file. Find the gid using nmi_runid2gid: 
    $ nmi_runid2gid <runid>
    The full directory path on nmi-s006 is /nmi/run/cndrauto/<year>/<month>/<gid>. The purpose of opening up access to the release directory is for other Metronome users to take the release binaries as inputs to their builds, tests or packaging runs.

  5. Make sure we've got a win32 build and releasable package (at this point, the nightly windows builds still don't provide packaging, so this remains a manual step).
    • Verify the contents of the .ZIP file
      • Check the layout of the file (currently there is an extra directory)
      • Check to see that condor_mail.exe is not missing.
      • Check for debug binaries 
        link /dump /imports bin\* | grep -e "MSVC.*D"
        If any of the condor programs or dlls are linked with the debug c-runtime, there will be output. Otherwise there will not be.
    • Either TJ or Scot can create the Windows MSI
    • Rename the zip and MSI file to include the platform string
    • Test the MSI installer.

  6. Deploy binaries onto production pool, start the "N-day" counter. Stable releases go onto the CS Dept. pool. Upgrading this pool is a complicated process in itself, which is documented on the Upgrading the UW CS Pool page. Developement releases go onto the BaTLab and CHTC pools. Talk to the infrastructure team to arrange upgrading these pools.
    • N >= 1 for a development series release
    • N >= 3 for a bug fix release in a stable series (X.Y.>0)
    • N >= 28 for a new stable series (X.Y.0)

  7. RUST-watcher must be diligent about looking for "bad things", bug reports from our users, etc.

  8. Verify that the Version History in the manual is up to date. PreparingVersionHistory

  9. Verify the list of supported platforms in the manual. This generally means talking to Pete Keller. The list is in doc/overview/overview.tex under 'Availability'. Furthermore, the platform-specific notes in the manual doc/platforms/* should be verified and updated if there have been any changes.

  10. Tag Build ID w/ release tag. For example, to create "V7_0_3" as opposed to "BUILD-V7_0_3-branch-2008-6-15":
    • (Assume a clone of CONDOR_SRC.git is present.)
    • cd CONDOR_SRC
    • git tag V7_0_3 BUILD-V7_0_3-branch-2008-6-15
    • git push origin V7_0_3 (This puts the V7_0_3 tag in the central CONDOR_SRC.git repo.)
    • Note: Push the TAG name, not the branch name.
    • update GitBranchDescriptions to document the new branch
      • Name of the tag and date the tag was created.
      • The branch the tag was created on.
      • The official version string included in that release.
      • The original build tag that the release tag was created from. This last point is very important, since it contains the actual date the release's codebase came from, not just the date the release tag "alias" was created. This is important for date-based comparisons in git, for example.
    • For example, here's the 6.9.5 tag's entry: 
      11/28/07 V6_9_5
  The offical tag for 6.9.5, from the V6_9_5-branch.
  Version string: $CondorVersion: 6.9.5 Nov 28 2007 BuildID: 65347 $
  Original build tag: BUILD-V6_9_5-branch-2007-11-28
      Note that in this case, everything was actually from the same date, but that's an exception. Usually, the date in the version string and original build tag will differ from the date the release tag itself was added.

Preparing to release to the world

  1. Release binary tarballs to the web. Releasing the binaries to the public is a two step process. First, we put the binaries into AFS at /p/condor/public/binaries/vX.Y/X.Y.Z. Then, we invoke the download system scripts to push these binaries from AFS out to the mirrors (this is a later step in these instructions).
    • WINDOWS: With <Current version +1> hopefully Windows packages will be automatic. If you have only the tarballs and not the Windows MSI/ZIP, you can build them from the tarballs. People who know how to make Windows packages: TJ, Cathrin, Z, Todd T.
      1. Unzip/untar the windows tarball into a working directory, c:\scratch\temp for instance. You will end up with all of the files in c:\scratch\temp\public
      2. Create a directory for the packagages, c:\scratch\temp\package for instance.
      3. Set your git repository to the branch that you are releasing and make sure that it doesn't have any extraneous files. 
        > cd CONDOR_SRC\msconfig
> dopackaging c:\scratch\temp\public c:\scratch\temp\package
        this will make the .msi and .zip for Windows in the c:\scratch\temp\package directory.

    • UNIX: Mostly this involves copying everything off of nmi-s006 and into /p/condor/public/binaries/...
      • Login to nmi-s006, then do the following (make sure to substitute for X.Y, X.Y.Z, <DATE> and <GID>) 
        # Make a new directory for this release
$ mkdir /scratch/release-X.Y.Z-<DATE>
$ cd /scratch/release-X.Y.Z-<DATE>
# Use the GID to find the tarballs.  First, verify the set of tarballs
$ ls -l /nmi/run/cndrauto/201?/*/<GID>/userdir/*/results.tar.gz
# Then unpack them into the current directory
$ find /nmi/run/cndrauto/201?/*/<GID>/userdir -type f -name results.tar.gz | xargs -I {} tar zxf {}
# Then transfer to AFS.  This can take awhile
$ cd public
$ scp *.deb *.rpm *.tar.gz tonic:/p/condor/public/binaries/vX.Y/X.Y.Z

  2. Pre-release source tarball. The source tarball used to be created every night as part of the nightly builds. After the conversion to cmake, it is no longer created. While releasing 7.5.5, I created it by hand by tarring the contents of userdir/common in the rundir of the NMI build. I put these files in a top-level directory named condor-X.Y.Z and I removed the soar directory. The source should be placed in /p/condor/public/binaries/vX.Y/X.Y.Z using the name "condor_src-X.Y.Z-all-all.tar.gz".

  3. Is this a new series release (x.x.0)? There are some additional details:

    • Add a new license agreement. This is usually just copying the old license to a new name and changing the version number. 
      cvs -d /p/condor/repository/HTML co condor-web
cd condor-web/src/downloads
cp v7.3.license.html v7.4.license.html
edit v7.4.license.html # change the Condor version numebr.
cvs add v7.4.license.html
cvs ci
cd ../..
./generate_html src

    • The download log files must exist; the download scripts won't create them if they don't. 
      cd /p/condor/public/license
touch licensers-v7.4 problem-v7.4 sendfile-v7.4

  4. Set the release date (the date the new version hit the Condor web site) in the Condor Manual's Version History item devoted to identifying the release date. OR, tell Karen exactly what date to use, and she'll do it for you. (#1236)

  5. Build the manual. 
    PATH=/s/gs/bin:$PATH
git clone /p/condor/repository/CONDOR_SRC.git
cd CONDOR_SRC
git checkout -b V7_0_3-branch origin/V7_0_3-branch
cd doc
make release

Release Condor to the world

Once you start following these steps, Condor is available to the public.

  1. Update the release date in the Google calendar.

  2. Release the manual. Move the symlink in /p/condor/public/html/manual for whatever release series you're changing to point at the new version. 
    cd /p/condor/public/html/manual
rm v7.0
ln -s v7.0.3 v7.0

  3. Tell the new download system about the new binaries.
    • All of the programs listed here take --help options, and the are all discussed to some extent in /p/condor/downloads/doc/operation.txt .
    • These tools require Python version 2.5. The /usr/bin/python on RHEL5 2.4.3, and these scripts won't run with it. Here is how to get Python 2.5 on CSL machines: 
      $ python -V     # Note that this is a capital "V".
Python 2.4.1
$ export PATH=/s/python-2.5/bin:$PATH
$ which python
/s/python-2.5/bin/python
$ python -V
Python 2.5
    • For more complete documentation on the download tools, look at /p/condor/downloads/doc/operation.txt

    • If this is a new series:

      • Edit /p/condor/downloads/etc/downloads.conf and add a few lines. Search for the previous series and follow its example. In particular, as a "[release 7.4] section, using a previous release as a template. Also, for each "[mirror SOMETHING]" section, add a release_7_4 entry, using previous releases as a template.

      • Update the download system's information. 
        download-versions update --lic=http://parrot.cs.wisc.edu/v7.4.license.html \
  --inst=http://www.cs.wisc.edu/condor/manual/v7.4/3_2Installation.html 7.4.0

    • Add a version to the new download "database". In this case, we'll install version 7.0.2, and we'll copy the release information from 7.0.1, except for the release date. Then, we'll edit the 7.0.1 description to reflect that it's no longer the most recent. Finally, we'll blow away 7.0.0 entirely. These steps are required. For all of the below commands, I'm going to assume that you've added /p/condor/downloads/bin to your PATH.

      • List the current versions: 
        $ download-versions print
      • Create the 7.0.2 version: 
        $ download-versions --copy 7.0.1 --date 2008/06/10 add 7.0.2
      • Update the 7.0.1 description: 
        $ download-versions update --description "Recent Stable Release" 7.0.1

    • Install the binaries into the new download system. This step is required. 
      $ download-install -v --mode symlink 7.0.2 /p/condor/public/binaries/v7.0/7.0.2

    • Push the new binaries out to the local mirror (parrot). If we didn't specify the "--mirror parrot", it'd push to all mirrors, including INFN, and that takes quite a while. A push to parrot usually takes on the order of 50 minutes or so. The binaries will be automatically pushed to all mirrors every AM, but this allows us to get it to one mirror at least. If you're prompted for a password, enter the password of the condor account on CSL machines. 
      $ download-push -v --mode real --mirror parrot 7.0.2

    • Generate symlinks to the binaries & push them out to the local mirror. This is also done by cron, twice per hour, so, if you didn't do this step, it'd get done for you. However, it only takes a few minutes, so I included it here. 
      $ download-gen-symlinks -v --mirror parrot

  4. Verify the downloads work.
    1. Download the binaries for one platform. Unpack them. Verify condor_version runs and returns the expected result.
    2. Download the binaries for a different platform. Unpack them. Verify that they look right.
    3. Download the source. Unpack it. Verify that it looks right.

  5. Push native packages into the repositories.
    1. Find a machine with at least 3 GB scratch space
    2. Obtain the RUNID and the Condor's branch name ( stable | development )
    3. If a new platform (OS or Arch) is added, see this documentation
    4. Check the mapping file to see if release platforms are correct 
      cat /p/condor/public/html/yum/files/mapping.txt
    5. Invoke the script to download packages and update the repositories. Verify the list of platforms to release. The script will wait for confirmation; proceed by pressing enter. 
      /p/condor/public/html/yum/files/manage_repo.pl -i 24009 -s /scratch/kooburat -b stable

Downloading from runid: 24009
Using scratch folder: /scratch/kooburat/condor_repo
Put packages into stable repos

Using mapping file from: /p/condor/public/html/yum/files/mapping.txt
The following platforms will be processed:
DEB : x86_deb_5.0       -> lenny-i386
DEB : x86_64_deb_5.0    -> lenny-amd64
RPM : x86_rhap_5        -> rhel5-i386
RPM : x86_64_rhap_5     -> rhel5-x86_64
RPM : x86_rhas_3        -> rhel4-i686
RPM : x86_64_rhas_3     -> rhel4-x86_64

Enter to confirm or Ctrl^C to cancel

Announce the Release

Now the binaries are available for download. Condor is released. However, you need to promptly announce the release as well.

  1. Note that following announcements are very generic. For minor bug fixes or features, that's fine. But if there is a major bug fix, a security fix, or a major new feature, call it out in the announcement.

  2. Update the web HTML regarding this release
    • cvs -d /p/condor/repository/HTML co condor-web
    • cd condor-web
    • edit src/lib/news.mas (to add an announcement 'Whats New?')
      • Here is a sample announcement: 
            { date => 'January 22, 2008',
      title => 'Condor 7.0.0 released!',
      news => <<ENDNEWS
The Condor Team is pleased to announce the release of
Condor 7.0.0. This first release in a new stable series
includes all new features added in the 6.9 development
series.  See the
<a href="http://www.cs.wisc.edu/condor/manual/v7.0/8_3Stable_Release.html#sec:New-7-0-0">Version
History</a> for a complete list of changes.  Condor 7.0.0
binaries and source code are available from
<a href="http://www.cs.wisc.edu/condor/downloads/">our
Downloads page.</a>
ENDNEWS
    },
    • edit src/downloads/index.html (to fix the release-specific text on the main downloads page)
    • touch src/index.html (so the main page updates with the version information from the downloads/index.html page)
    • touch src/manual/index.html (so the manual links update with the version information from the downloads/index.html page)
    • NOTE: if you're doing the first release of a new release series, you'll also need to edit:
      • src/index.html to:
        • change the "The Condor Manual:" list of links (chances are good you'll want to remove something older when you add the new one -- we'll never want that list to be longer than 3 links).
      • src/developers/index.html to add the new series to the "Manuals and Other Documentation" list.
      • edit src/manual/index.html (to update which versions of each branch/release series are the current manual)
    • cvs commit (all of your changes in condor-web)
    • ./generate_html src (to update the HTML on the web with your changes -- note this only works on 32 bit machines, the 64 bit lab machines are missing some perl module or another).

  3. Send email to condor-world and condor-users. Here is a sample announcement: 
    Hello,

The Condor Team is pleased to announce the release of Condor 7.0.0. This first
release in a new stable series includes all new features added in the 6.9
development series.  See the Version History for a complete list of changes.
Condor 7.0.0 binaries and source code are available from our Downloads page.

Version History:
  http://www.cs.wisc.edu/condor/manual/v7.0/8_3Stable_Release.html#sec:New-7-0-0

Downloads Page:
  http://www.cs.wisc.edu/condor/downloads/


Thank you for your interest in Condor!


- The Condor Team

Post Release

These are just as important at the release process proper!

  1. Turn off the old downloads. These steps take a long time to run. Note that we typically want 5 different releases available for download: current stable (7.4.4), recent stable release (7.4.3), previous stable release (7.2.5), current developer release (7.5.4) and recent developer release (7.5.3). The "recent" releases are so people can revert if something terrible happens. The previous table release is for users who delay stable upgrades.
    • Disable the 7.0.0 (X.X.n-2) release. This prevents it from appearing in the download list. 
      download-versions up 7.0.0 --disable
    • Delete the 7.0.0 release. The first step "download-delete" will delete the files maintained by the download subsystem. The second step "download-versions delete" will delete the version from the version database. So, do the following: 
      $ download-delete 7.0.0
    • NRL says: Don't do this, at least for now.... 
      $ download-versions delete 7.0.0

  2. Archive release products to DVD. Get recordable DVDs from Ken (no sleeves!). Drop off the finished DVD with Pete, who will store it in a box in his filing cabinet. Include the following (the archiving directions include these):
    • Binary tarballs. This includes archiving the unstripped binaries.
    • Matching glidein tarballs. Get these from /p/condor/public/binaries/glidein. 
      $ ssh tonic "cd /p/condor/public/binaries; tar cf - glidein/7.1.3" | tar xvfpS -
    • Source tarball.
    • Manual: tarball of HTML tree, and ref.pdf. The HTML files are in /p/condor/public/html/manual. Do something like this: 
      $ ssh tonic "cd /p/condor/public; tar cf - html/manual/v7.1.3" | tar xvfpS -

  3. Stash the condor repository onto nmi-s006 by doing something like: 
    $ ssh nmi-s006
$ cd /scratch/condor_stash
$ sudo -u cndrauto ./stash_condor -v V7_4_1
    (Replace V7_4_1 with the git tag of whatever version you want the externals stashed for). Talk to Matyas if you have any problems/comments.
  4. On nmi-s006, unpin the previous release of Condor in this series by using /nmi/bin/nmi_pin. You can find runid for the previous release in this series of Condor at GitBranchDescriptions. Do not do this until you confirm that the previous Condor revision is archived to DVD. Example: If you are releasing Condor 7.0.1, then find the runid for 7.0.0 and unpin that when you know its DVD archival status. 
    $ nmi_pin --unpin --force <runid>

  5. Close out all tickets fixed by this release.

  6. Move unresolved tickets to the next release. ticket-target-mover, documented at ScriptingGitTrac may be assistance.

  7. Update the Condor Wikipedia page to indicate the current release.

  8. Stop building the branch nightly:
    • cd /p/condor/home/tools/cvs-tools
    • Edit create_build_tag-input
    • Delete the section that defined the nightly build tag for this branch.
    • cvs diff create_build_tag-input (to verify the change before you commit it)...
    • cvs commit -m "removed nightly build tags for V6_X_Y-branch" create_build_tag-input