HTCondorWiki: Bat Lab Build

This is for the new BATLab (submit-*.batlab.org). For the obsolete, pre-2012 BATLab (nmi-s00*.cs.wisc.edu), see NmiBuild.

Building a HTCondor workspace in NMI

Automatically from Linux

Try /afs/cs.wisc.edu/u/a/d/adesmet/bin/submit-workspace . Run it from the top of your workspace. You'll want to have public-key authentication configured for the current primary BaTLab submit host. You'll be asked for your private key password once at the start. The script will create a directory on the remote side named after the branch and the current date and time. The script does not delete the copy after it's no longer necessary; remember to clean up occasionally.

Just in case, the script is attached.

If things go wrong, fall back to "Manually from Linux," below. The script is basically implementing those steps.

Manually from Linux

Here is a common scenario: You would like to do a build on some platform or set of platforms in order to ensure that you haven't broken the build process. What are your options? You could check in the code and wait for the nightly builds, but you may well break the build and receive fifty lashes with a wet noodle. A better solution is to not check in your code, but use NMI to build it on any platform that you like.

Before you start, make sure that you have an account on batlab.chtc.wisc.edu.

From your usual machine,

  % ssh batlab.chtc.wisc.edu mkdir workspace
  % exit
  % cd to top of clone(CONDOR_SRC)
  % git write-tree
  % git archive xxxxxx(hash created by git write-tree) | ssh batlab.chtc.wisc.edu cd workspace '&&' tar xf -
For Windows execute below command,
  % git archive xxxxxx(hash created by git write-tree) | ssh USERNAME@batlab.chtc.wisc.edu cd workspace ; tar xf -

Then, log into batlab.chtc.wisc.edu, and run the following commands

  % cd workspace
  % cd nmi_tools
  % ./condor_nmi_submit --build --git --workspace=../ --platforms=all --notify-fail-only --desc="A Description"

If you're trying to build something older than 7.7, you'll need to do the following before the condor_nmi_submit. It's not necessary for 7.7 and later.

  % git --git-dir=/space/git/CONDOR_DOC.git archive origin/master | tar xf -

Some important options:

--without-tests suppresses the automatic submission of tests when the build is done.
--platforms takes a comma separated list of platforms. "all" is a good default. For a list of available platforms, see "Platforms" below. You must specify the platforms you want. (This is a change from the old BATLab.) You can find a list of available platforms at http://batlab.chtc.wisc.edu/nmi/pool/status , look for "Pool usage by platform" in the left column. At the moment (2014-10-30) the set of platforms that we do nightly and rolling builds on is:
- x86_64_Debian0,x86_64_Debian6,x86_64_Debian7,x86_64_Fedora19,x86_64_Fedora20,x86_64_MacOSX7,x86_64_MacOSX8,x86_64_RedHat5,x86_64_RedHat6,x86_64_RedHat7,x86_64_SL6,x86_64_Solaris11,x86_64_Ubuntu10,x86_64_Ubuntu12,x86_64_Ubuntu14,x86_64_Windows7,x86_64_Windows8,x86_RedHat5,x86_RedHat6,x86_SL5
--use-externals-cache uses cached externals to speed the build. May not work in new BATLab. If you're testing a new external, do not use this option.
--release-only builds the dynamic binaries, but doesn't package or return them. If you're just doing a compile test and don't need the binaries, this will speed the build by about 45 minutes. If you need to generate a package to give to a user (or to submit for testing), omit this option.
--desc specifies a description that appears in the NMI web pages and can help identify your build.
--workspace=../ is the location of your workspace. Warning: due to a bug in condor_nmi_submit, as of 2006-02-14 the only valid option is "../" as given in the example.
--append 'append_requirements=Machine!="exec3.batlab.org"' Blacklist exec3 from the hosts you will build on. {endcode}

The script has a couple of other options. run condor_nmi_submit without options for a summary.

Checking your build's status

HTCondor-specific status page (Maintained by at (Maintained by Scot K. as of 2012-01).
Metronome status page (Maintained by Todd M. as of 2012-01.)

Attachments:

submit-workspace 1821 bytes added by adesmet on 2014-Nov-18 19:30:15 UTC.
Script to automatically submit a workspace build to BaTLab