This is all for the old BATLab (nmi-s00*.cs.wisc.edu). For the new BATLab (submit-*.batlab.org), see BatLabBuild. {section: Building HTCondor in NMI} {subsection:Resubmitting last night's builds, or one-off nightlys} See ResubmitLastNightsBuilds for a one-off reusing last night's tag, or using a new tag. {subsection: Building a branch in NMI} Significant new features should be committed into the git repository on their own branch, and this branch should successfully build and pass all regression tests on all supported platforms _before_ the code is merged back into the master branch. *WARNING* As of 2010-05-12, NMI pulls its repository from the public HTCondor repository (http://condor-git.cs.wisc.edu/repos/condor.git). As such, it only includes public branches. Thus, random other working branches will not generally be available, and the following _will not work_. *Step-by-step instructions:* 1: Log into nmi-s006.cs.wisc.edu (get an account if necessary). 1: Become the cndrauto user. Hint: you probably have sudo access, from there =su - cndrauto=. 1: As the cndrauto user, run git fetch to update the git repo on nmi-s006.cs.wisc.edu from the latest copy on the central repository. Make sure that /prereq/bin (where git is found on nmi-s006.cs.wisc.edu) is in your PATH. {code} % sudo su - cndrauto % git --git-dir=/space/git/CONDOR_SRC.git fetch {endcode} If you just made a tag or pushed a commit you want to build, check that it has made it through the pubic git repository to the local one. {code} % git --git-dir=/space/git/CONDOR_SRC.git log origin/V7_4-branch {endcode} If you don't see your tag or commits, wait a few minutes, then retry the git fetch command. 1: Exit back to your real login. {code} % exit {endcode} 1: Submit the build as yourself. Assuming the branch you want to build is _V7_4-branch_, you'd do the following: *: Checkout the nmi_tools associated with the branch/tag: {code} % mkdir -p /tmp/ % cd /tmp/ % rm -rf ./nmi_tools # If you want to build a specific tag, omit the following origin/ and just # use the tag name. % /prereq/git-1.5.4/bin/git archive --remote=/space/git/CONDOR_SRC.git \ origin/V7_4-branch nmi_tools | tar xv {endcode} *: Submit the build. {code} % ./nmi_tools/condor_nmi_submit --build --git \ --desc="Testing my fix to Foo" \ --module=UNUSED --use-externals-cache \ --src-tag=V7_4-branch \ --other-tag=V7_4-branch {endcode} *Some notes:* *: To build the trunk, use the src and other tag "master". (This is relatively obvious, but is mentioned because it was not nearly so obvious with CVS.) *: The =--src-tag= describes your branch; the =--other-tag= is the branch to use to pull the docs from. You do not need the 'origin/' specifier when using =--src-tag= and =--other-tag= in this manner. If you wanted to get the source and manual from the same branch, then use only --tag=origin/. *: If you don't want builds on all supported platforms, use option =--platforms=. *: Note the above will submit both a build and a test. *: If you want to submit a new "nightly" build for a given day, a couple things are different: *:: Create and push a new build tag on the appropriate branch of both CONDOR_SRC and CONDOR_DOC. Take the most recent tag and add {quote:_2}. {code} % cd CONDOR_SRC % git tag BUILD-V7_5_4-branch-2010-10-17_2 % git push origin BUILD-V7_5_4-branch-2010-10-17_2 % cd ../CONDOR_DOC % git tag BUILD-V7_5_4-branch-2010-10-17_2 % git push origin BUILD-V7_5_4-branch-2010-10-17_2 {endcode} *:: Update both /space/git/CONDOR_SRC.git and /space/git/CONDOR_SRC.git on nmi-s006. *:: Submit the build as user cndrauto with the nightly flags: {code} % cd ~cndrauto/condor % nmi_tools/condor_nmi_submit --build --nightly --use-externals-cache \ --submit-xtests --notify='condor-builds@cs.wisc.edu' \ --notify-fail-only --git --tag=BUILD-V7_5_4-branch-2010-10-17_2 \ --module=UNUSED {endcode} *: =--use-externals-cache= is ignored for Cmake builds (V7_5_5), cached externals are always disabled for V7_5_5 until V7_7_0. For V7_7_0 and later, cached externals are always enabled. With V7_7_0 you can use =--clear-externals-cache= to clear the cache before the build. *: The blacklist a particular host, add something like this to the condor_nmi_submit line: {code} --append 'append_requirements=Machine!="nmi-0102.batlab.cs.wisc.edu"' {endcode} {subsection: Building a HTCondor workspace in NMI} Here is a common scenario: You would like to do a build on some platform or set of platforms in order to ensure that you haven't broken the build process. What are your options? You could check in the code and wait for the nightly builds, but you may well break the build and receive fifty lashes with a wet noodle. A better solution is to *not* check in your code, but use NMI to build it on any platform that you like. Before you start, make sure that you have an account on nmi-s006.cs.wisc.edu. You also need to be in the cndr-cvs group. NMI staff can set you up with both. This assumes that the changes you want to test are checked into your local git repo in a branch named =feature-branch=. (Another option is to scp your workspace to nmi-s006.cs.wisc.edu.) Here's how it works: From your usual machine, {code} % ssh nmi-s006.cs.wisc.edu mkdir workspace % cd path-to-git-dir % git archive feature-branch | ssh nmi-s006.cs.wisc.edu cd workspace \&\& tar xf - {endcode} If you don't have a feature-branch and want to build your workspace as is, use the following. {code} % ssh nmi-s006.cs.wisc.edu mkdir workspace % cd path-to-git-dir % tar cf - * | ssh nmi-s006.cs.wisc.edu cd workspace \&\& tar xf - {endcode} Then, log into nmi-s006.cs.wisc.edu, and run the following commands {code} % PATH=/prereq/bin:$PATH % cd workspace # Note: this next git line is no longer necessary on the current (7.7.x) master % git --git-dir=/space/git/CONDOR_DOC.git archive origin/master | tar xf - % cd nmi_tools % ./condor_nmi_submit --build --git --workspace=../ --notify-fail-only --use-externals-cache --desc="A Description" {endcode} Note that you probably only need the git archive operation the first time you do this. Some important options: *: =--without-tests= suppresses the automatic submission of tests when the build is done. *: =--platforms= takes a comma separated list of platforms. Omitting it will build your workspace on all platforms HTCondor supports. For a list of available platforms, see "Platforms" below. *: =--use-externals-cache= uses cached externals to speed the build. If you're testing a new external, do *not* use this option. *: =--release-only= builds the dynamic binaries, but doesn't package or return them. If you're just doing a compile test and don't need the binaries, this will speed the build by about 45 minutes. If you need to generate a package to give to a user (or to submit for testing), omit this option. *: =--desc= specifies a description that appears in the NMI web pages and can help identify your build. *: =--workspace=../= is the location of your workspace. *Warning:* due to a bug in condor_nmi_submit, as of 2006-02-14 the only valid option is "=../=" as given in the example. *: =--append 'append_requirements=Machine!="nmi-0102.batlab.cs.wisc.edu"'= Blacklist nmi-0102 from the hosts you will build on. {endcode} The script has a couple of other options. run condor_nmi_submit without options for a summary. {subsection: Platforms} If you don't know what platforms to use, you may view the {link: http://nmi-s006.cs.wisc.edu/nmi/?page=pool/index NMI pool status page} (select a host link for more information about installed prerequisite software) or use the /nmi/bin/nmi_condor_status command. It's best to make your terminal window wide and use the -ww option to get the most output. For example: % nmi_condor_status -ww vm1@nmi-build14.cs.wisc.edu | ppc_aix_5.2 | Claimed | Busy | 06:05:338 | cndrauto@cs.wisc.ed| 11358 vm2@nmi-build14.cs.wisc.edu | ppc_aix_5.2 | Claimed | Busy | 02:11:690 | cndrauto@cs.wisc.ed| 11360 nmi-hpux | hppa_hpux_B.10.2| Unclaimed | Idle | 02:15:906 | (None) | (None) vm1@nmi-build34.cs.wisc.edu | irix_6.5 | Unclaimed | Idle | 03:45:272 | (None) | (None) vm2@nmi-build34.cs.wisc.edu | irix_6.5 | Unclaimed | Idle | 03:45:270 | (None) | (None) vm1@nmi-irix.cs.wisc.edu | irix_6.5 | Unclaimed | Idle | 02:05:323 | (None) | (None) vm2@nmi-irix.cs.wisc.edu | irix_6.5 | Unclaimed | Idle | 02:05:309 | (None) | (None) ... vm2@nmi-build29.cs.wisc.edu | sun4u_sol_5.8 | Claimed | Busy | 1+04:31:191 | cndrauto@cs.wisc.ed| 11328 vm1@nmi-0050.cs.wisc.edu | sun4u_sol_5.9 | Unclaimed | Idle | 35:21 | (None) | (None) vm2@nmi-0050.cs.wisc.edu | sun4u_sol_5.9 | Unclaimed | Idle | 35:21 | (None) | (None) nmi-build18 | sun4u_sol_5.9 | Claimed | Busy | 06:26:157 | cndrauto@cs.wisc.ed| 11357 vm1@nmi-solaris10.cs.wisc.edu| sun4u_sol_5.9 | Claimed | Busy | 11:04:270 | cndrauto@cs.wisc.ed| 11350 vm2@nmi-solaris10.cs.wisc.edu| sun4u_sol_5.9 | Claimed | Busy | 15:18:113 | cndrauto@cs.wisc.ed| 11336 janet.cs.wisc.edu | x86_winnt_5.1 | Unclaimed | Idle | 03:34:209 | (None) | (None) {subsection: Testing a HTCondor NMI workspace build in NMI} Assuming you're still in the nmi_tools subdirectory of your workspace, and that your build with runid 1234 is completed, you can submit a test run with the following: % ./condor_nmi_submit --test --platforms=all \ --buildid=1234 \ --desc="Description of this build" --workspace=../ {subsection: Checking your build's status} The NMI job status page is at http://nmi-s006.cs.wisc.edu/nmi/?page=results/overview. There's a HTCondor-specific page at http://nmi-s006.cs.wisc.edu/results/Run-condor.php. {subsection: Extracting your binaries} To extract the binaries from the builds, use =nmi_tools/condor_nmi_extract=. The script writes to =./cache= and =./public= -- and will create these directories if they don't already exist. The cache is to maintain a list of things that it's already extracted from, so if you run it again, it won't do unnecessary work. Run the python in /prereq/Python-2.6.2/bin (put it in the front of your path). Use =condor_nmi_extract -h= to get help.