Page History
- 2020-Jan-22 11:53 tim
- 2015-Sep-28 17:55 johnkn
- 2015-Sep-28 17:54 johnkn
- 2012-Nov-13 15:31 adesmet
- 2012-Aug-15 16:14 kronenfe
- 2012-Aug-15 15:44 kronenfe
- 2012-Aug-15 15:38 kronenfe
- 2012-Aug-15 12:40 kronenfe
- 2012-Aug-15 12:39 kronenfe
- 2011-Apr-07 14:42 kronenfe
- 2011-Apr-07 14:42 kronenfe
- 2011-Apr-07 14:34 gthain
- 2011-Apr-07 14:24 kronenfe
Overview
We build HTCondor and run the regression test suite on each push to the Git repository. The reasons for these frequent builds are:- Provide quick feedback if a commit breaks the builds or tests
- Provide historical coverage to help trace back and find which commit broke tests/builds. It is hard to track backwards from a broken test to the commit which broke the test if there are many commits between builds. The more frequent the tests are, the smaller the commit windows is to find the guilty commit.
- Help spot race conditions in tests that fail sporadically, especially when subsequent runs of the test produce different results against the same code
Current implementation
The per-commit runs are submitted from batlab.chtc.wisc.edu in the /home/condorauto/per-commit directory. A single Crondor job exists which wakes up every few minutes and checks for new commits to build. This job simply calls condor_nmi_submit to do the dirty work. Occasionally, someone will condor_rm all of condorauto's jobs, forgetting about this Crondor job. When that happens, the Crondor job will get removed too, stopping the per-commit builds from running (but don't worry - it won't lose state) and someone needs to re-submit the job (see below for instructions).
Note that condor_nmi_submit sets the JobPrio such that the most recently submitted build (and test) has the highest priority. Because tests are slow, they usually fall behind in the course of a day, but the most recently submitted ones are started first. Even if it takes overnight for a test run to finish, it can still be very valuable when tracking backwards to find what commit broke a test.
Setting up the per-commit builds
The per-commit build "code" is in Subversion at /p/condor/software/svn/condor/per-commit-builds
Follow the README file found in that directory.
[[anonymous: TJ - I think it's in git now?]]
Re-submitting the per-commit builds
- Login to
batlab.chtc.wisc.eduand sudo to thecondorautouser cd /home/condorauto/per-commitcondor_submit submit
We-winding the per-commit builds for a branch
- Login to
batlab.chtc.wisc.eduand sudo to thecondorautouser cd /home/condorauto/per-commitgit --git-dir=/home/condorauto/condor.git tag continuous-build-placeholder-tag-master --force master^
replace master with the name off the branch you want to rewind.
Per-commit Native Package Repositories
The per-commit builds generate RPMs and Debian packages. These are placed into repositories on mirror.batlab.org.
YUM repository Debian repository
As of this writing (2012-08-15) there are no known external users, although Peter Couvares has expressed interest in getting pre-releases from here.
TODO list:
- Make this work for Debian - it only works for yum currently because I did not have time to research the Debian native package directory structure.
- Generate .repo files automatically for each YUM repo
- Make a nicer front page with all the .repo files?
- Decide how many RPMs and Debs to keep around at a time
The repositories are kept up-to-date through a combination of the build glue which runs on the submit node (batlab.chtc.wisc.edu) and a script that runs on mirror.batlab.org. The glue pushes all RPMs (and eventually Deb files too) to a well known directory on mirror.batlab.org. The script on mirror.batlab.org runs out of cron and wakes up periodically, looks in the well known directory, moves files into their respective repo (creating the directories if needed), and re-runs createrepo (or the debian equivalent).
The code is available in SVN at: /p/condor/software/svn/condor/testing-repo-scripts
The Build and Test Dashboard
The code is in PHP. It is available in the same SVN repository as the per-commit build scripts: /p/condor/software/svn/build-dashboard/
Currently it is made available on batlab.chtc.wisc.edu by placing it in /home/condorauto/public/html/results and placing a symlink in /var/www/html
[kronenfe@batlabsubmit0001 html]$ ll /var/www/html ... lrwxrwxrwx 1 root root 36 Sep 8 2011 results -> /home/condorauto/public/html/results lrwxrwxrwx 1 root root 42 Sep 22 2011 results-devel -> /home/condorauto/public/html/results-devel ...