Overview
HTCondor currently provide native packages in Debian and RPM format. Location of the repositories are listed below:http://www.cs.wisc.edu/condor/yum
http://www.cs.wisc.edu/condor/debian
Background
Providing package via repository offers many advantages. Firstly, many OS distributions automatically detect package change and notify users to update. Secondly, package management programs such as yum or apt-get automatically resolve dependency
Repository Layout
There are 3 separate repositories: stable, development and testing (pre-release testing repo).
Yum
/p/condor/public/html/yum/stable/rhel5/*.rpm /p/condor/public/html/yum/stable/rhel4/*.rpm /p/condor/public/html/yum/development/rhel5/*.rpm /p/condor/public/html/yum/development/rhel4/*.rpm
Debian
/p/condor/public/html/debian/stable/dists/lenny/contrib/binary-i386/*.deb /p/condor/public/html/debian/stable/dists/lenny/contrib/binary-amd64/*.deb ... /p/condor/public/html/debian/development/dists/etch/contrib/binary-i386/*.deb /p/condor/public/html/debian/development/dists/etch/contrib/binary-amd64/*.deb ...
Manually Managing Repositories
Yum
Add/remove packages
- Go to a specific branch such as /p/condor/public/html/yum/stable/rhel5/ and add/remove package from the directory.
- Invoke createrepo to update the repo index.
createrepo /p/condor/public/html/yum/stable/rhel5/
Add a new OS distribution
- Create new branch such as /p/condor/public/html/yum/stable/rhel6/ and follows the same step as add/remove package.
Debian
Add/remove packages
- Add/remove package from its branch such as /p/condor/public/html/debian/stable/dists/lenny/contrib/binary-i386
- Set appropriate environment variable for invoking repository utility
export PATH=$PATH:/p/condor/public/html/debian/stable/dists/lenny/contrib/binary-i386/ export PERL5LIB=$PERL5LIB:/p/condor/workspaces/vdt/debian-build-tools/lib/perl5/vendor_perl/5.8.8
- Go to the target folder and invoke dpkg-scanpackages to generate index file
dpkg-scanpackages --arch i386 --multiversion . "/dev/null" > Packages
- Create compressed version of index file (apt-get or synaptic will download only the compressed version)
gzip -9c Packages > Packages.gz bzip2 -9k Packages
Add a new OS distribution
- Copy entire distribution for existing one
cp -r /p/condor/public/html/debian/stable/dists/lenny /p/condor/public/html/debian/stable/dists/squeeze
- Remove all packages (.deb) and index files (Packages.gz, Packages.bz)
- Edit archive tag in the Release file. Possible values are: stable, oldstable, testing or unstable. Release file is used for APT pinning.
- Follows Step 2 of add/remove packages
Automate Script for Managing Repositories
Related Files
| File | Description |
| /p/condor/public/html/yum/files/manage_repo.pl | Script |
| /p/condor/public/html/yum/files/mapping.txt | Config file |
Overview
The main goal of this script is to facilitate the release process. The script accepts NMI RUNID as a parameter. It uses the same logic as nmi_crowbar to query URLs from NMI database of a given RUNID. Then it will read the mapping.txt which provides mapping between NMI platform and platform name on each repo in order to download packages and put them into the correct location. Finally, it will update the repositories' index files.
In normal mode of operation 1-2 GB should be enough for scratch disk requirement because it only download and extract one package at time.
mapping.txt Format
| NMI_PLATFORM | TYPE | ARCH | TARGET_OS |
| ia64_rhas_3 | RPM | ia64 | rhel4 |
| x86_deb_5.0 | DEB | i386 | lenny |
- The first column specifies a NMI platform name.
- Type must be DEB or RPM
- ARCH is the architecture string.For RPM, it should match the arch string in RPM's name. For Debian, it must match Debian-style arch string (i386, ia64, alpha, amd64, .. )
Note that we use packages built on RHEL3 on RHEL4. The script will only download packages from platforms listed in the mapping.
The script is not capable of add a new OS distribution via mapping.txt. Please use the manual method to add a new OS distribution into a repository before add new mapping in mapping.txt
Script Usage
- To download packages from a given RUNID, use /scratch/kooburat as scratch folder and push them into the stable repos. It will also update the repos when finishes.
manage_repo.pl -i 24009 -s /scratch/kooburat -b stable
- To download packages from a given RUNID into scratch folder.
manage_repo.pl -i 24009 -s /scratch/kooburat
- To remove a specific version of package from a given set of repos.
manage_repo.pl -b development -r 7.3.1-1
Using Repositories
Stable and Development Repositories
Please see below for detailed instructions
http://www.cs.wisc.edu/condor/yum
http://www.cs.wisc.edu/condor/debian
Internal Testing Repositories
To test pre-release packages. Testing repositories are created for both Debian and Yum repository. Please use the instruction below to access these repositories.
Note: Access to these repositories is limited only to wisc.edu network.
Yum
Use the following repo files (Depends on platform)
http://www.cs.wisc.edu/condor/yum/repo.d/condor-testing-rhel4.repo http://www.cs.wisc.edu/condor/yum/repo.d/condor-testing-rhel5.repo
Debian
Add the following lines to /etc/apt/sources.list (Depends on platform)
deb http://www.cs.wisc.edu/condor/debian/testing/ etch contrib deb http://www.cs.wisc.edu/condor/debian/testing/ lenny contrib
Download Statistics
We use CSL's facility to capture the download statistic on these repositories. A weekly statistic email will be sent to the addresses listed in these files.
/p/condor/public/html/yum/.statinfo
/p/condor/public/html/debian/.statinfo
Please see below documentation for more detail.
http://www.cs.wisc.edu/twiki/bin/view/CSDocs/WebServerFAQs#How%20Do%20I%20Get%20Statistics%20For%20My%20W