Page History
- 2020-Jan-22 11:46 tim
- 2015-Nov-06 11:52 johnkn
- 2015-Oct-07 15:46 johnkn
- 2015-Oct-07 14:16 johnkn
- 2015-Oct-07 13:41 johnkn
- 2015-Oct-07 13:20 johnkn
- 2015-Oct-01 14:26 johnkn
- 2015-Oct-01 14:18 johnkn
- 2015-Oct-01 14:12 johnkn
- 2015-Oct-01 13:43 johnkn
- 2015-Oct-01 13:25 johnkn
- 2015-Oct-01 13:12 johnkn
- 2015-Oct-01 13:12 johnkn
- 2015-Oct-01 13:11 johnkn
TJ's simple scripts
sort_ads.pl
A script to sort a collection of classads by key without reordering the ads. What it actually does is accumulate lines from stdin until it gets to a blank line, then prints out those lines sorted.sort_ads.pl
#!/usr/bin/env perl
use strict;
use warnings;
my %ad;
for my $line ( <STDIN> ) {
if ($line =~ /^\s*$/) {
# print out the previous add sorted by key then reset the hash for a new ad.
if (%ad) {
for (sort keys %ad) { print $_; }
undef %ad;
}
}
$ad{$line} = 1;
}
# print the last ad sorted by key
if (%ad) {
for (sort keys %ad) { print $_; }
undef %ad;
}
condor_status -long | perl sort_ads.pl
Scripts for building on Windows
cmakeit.bat
run this script to create a condor.prj file in the current directory. Does much the same thing as configure_uw, but works on Windows. If sources are not in the current directory, it depends on the environment variable GIT_HOME to find them. This script will refuse to run if the current directory is a git repo.
Optional arguments are
- regen - don't suppress cmake regeneration
- analyze - enable MSVC static code analysis for the build (take 3x longer)
- buildid <id> - set the build id
cmakeit.bat
@echo off setlocal set _noregen=-DCMAKE_SUPPRESS_REGENERATION:BOOL=TRUE if "%1"=="regen" ( shift set _noregen=-DCMAKE_SUPPRESS_REGENERATION:BOOL=FALSE ) if "%1"=="analyze" ( shift set _analyze=-DMSVC_ANALYZE:BOOL=TRUE ) if "%1"=="noanalyze" ( shift set _analyze=-DMSVC_ANALYZE:BOOL=FALSE ) if "%1"=="tests" ( shift set _test=-DBUILD_TESTING:BOOL=TRUE -DWITH_BOOST:BOOL=TRUE ) if "%1"=="buildid" ( set _buildid=-DBUILDID:STRING="%~2" shift shift ) if exist .\.git\config goto :nothere if exist .gitignore goto :nothere if exist .\src\condor_utils\condor_api.h ( set _src= ) else ( set _src=%GIT_HOME% choice /M "no sources in current dir, use %GIT_HOME%" if errorlevel 2 goto :EOF ) set _gen="Visual Studio 11" if "%VS_DIR%"=="C:\vs90" set _gen="Visual Studio 9 2008" REM set _gen="NMake Makefiles" if NOT ~%1==~ set _gen=%1 @echo on cmake -LA CMakeLists.txt -G %_gen% %_noregen% %_analyze% %_test% %_buildid% -D_VERBOSE:BOOL=TRUE %_src% @goto :EOF :nothere @echo You don't want to run this when the current directory is a git repository.
copy cmakeit.bat d:\scratch\build set GIT_HOME=d:\scratch\master\CONDOR_SRC mkdir d:\scratch\build\master cd d:\scratch\build\master ..\cmakeit
rebuild.bat
run this script while in your build directory (the one with condor.sln) to build and/or release HTCondor. Optional arguments are
- build - do minimal build, default is full rebuild
- con|console - send output to console, default is to send it to rebuild.out
- debug|retail|RelWithDebInfo - set build type, default is retail
- package|externals|install - build the given target. ALL_BUILD is the default build target.
- release [test] - after build, invoke the release.bat script, which will release into release_dir subdir unless the test argument is passed, then it will release into the test subdir.
rebuild.bat
@echo off
setlocal
for %%I in (wget.exe) do if "%%~f$PATH:I"=="" (
echo WGET is not in the path - aborting
exit /b 1
)
if not exist condor.sln (
echo There is no Condor.sln file in the current directory
echo did you forget to run cmakeit.bat?
exit /b 1
)
set _scriptdir_=%~sdp0
set _counterrs_=call %~sdp0counterrs.bat
:top
if "~%1"=="~" goto doit
goto %1
:debug
set _bldtype_=Debug
goto next
:con
:console
set _console_=true
goto next
:build
set _rebuild_=Build
goto next
:relwithdebinfo
:retail
set _bldtype_=RelWithDebInfo
goto next
:package
set _target_=PACKAGE
goto next
:externals
set _target_=ALL_EXTERN
goto next
:install
set _target_=INSTALL
goto next
:release
set _release_=call %_scriptdir_%release.bat
if "%2"=="test" (
set _release_=%_release_% test
shift
)
if "%_bldtype_%"=="Debug" (
set _release_=%_release_% debug
)
goto next
:next
shift
goto top
:doit
if ~%_rebuild_%==~ set _rebuild_=Rebuild
if ~%_bldtype_%==~ set _bldtype_=RelWithDebInfo
if ~%_target_%==~ set _target_=ALL_BUILD
set _counterrs_=call %_scriptdir_%counterrs.bat %_rebuild_%.out
::@echo '%_counterrs_%'
@if NOT "%_console_%"=="true" goto :build_to_log
@echo on
devenv condor.sln /%_rebuild_% %_bldtype_% /project %_target_%
@echo off
goto :EOF
:build_to_log
@echo on
devenv condor.sln /%_rebuild_% %_bldtype_% /project %_target_% > %_rebuild_%.out
@echo off
%_counterrs_%
%_release_%
copy rebuild.bat d:\scratch\build cd d:\scratch\build\master mkdir d:\scratch\build\master\test ..\rebuild build debug release test
counterrs.bat
This script processes the build.out or rebuild.out log produced by rebuild.bat and prints a summary of errors and warnings.
counterrs.bat
@echo off
@setlocal
set infile=%1
if "%infile%"=="" set infile=Rebuild.out
set tmpfile=%infile:.out=%_nopre.tmp
:: extract a sorted list of error/warning messages without the "nn>" prefix on the lines.
grep -E "C[0-9]{4}:" %infile% | awk "{gsub(/^[0-9]+>/,\"\")};1" | sort > %tmpfile%
:: set variables to conain counts of various things.
for /F %%I in ('grep -E " [C|E][0-3][0-9]{3}" %infile% ^| sort ^| uniq ^| awk "END{print NR}"') do set errs=%%I
for /F %%I in ('grep -E " C4[0-9]{3}" %infile% ^| sort ^| uniq ^| awk "END{print NR}"') do set warns=%%I
for /F %%I in ('grep -E " C6[0-9]{3}" %infile% ^| sort ^| uniq ^| awk "END{print NR}"') do set anals=%%I
for /F %%I in ('grep -E " C6[0-9]{3}" %infile% ^| grep -I -E "\.h" ^| sort ^| uniq ^| awk "END{print NR}"') do set analhs=%%I
set analmsg=
if NOT "%anals%"=="0" set analmsg=%anals% Analysis warnings
set analhmsg=
if NOT "%analhs%"=="0" set analhmsg=^(%analhs% in headers)
@echo %infile% Raw : %errs% Errors, %warns% Warnings %analmsg% %analhmsg%
for /F %%I in ('grep -E " [C|E][0-3][0-9]{3}" %tmpfile% ^| sort ^| uniq ^| awk "END{print NR}"') do set errs=%%I
for /F %%I in ('grep -E " C4[0-9]{3}" %tmpfile% ^| sort ^| uniq ^| awk "END{print NR}"') do set warns=%%I
for /F %%I in ('grep -E " C6[0-9]{3}" %tmpfile% ^| sort ^| uniq ^| awk "END{print NR}"') do set anals=%%I
for /F %%I in ('grep -E " C6[0-9]{3}" %tmpfile% ^| grep -I -E "\.h" ^| sort ^| uniq ^| awk "END{print NR}"') do set analhs=%%I
set analmsg=
if NOT "%anals%"=="0" set analmsg=%anals% Analysis warnings
set analhmsg=
if NOT "%analhs%"=="0" set analhmsg=^(%analhs% in headers)
@echo %infile% Uniq : %errs% Errors, %warns% Warnings %analmsg% %analhmsg%
if not "%errs%"=="0" @grep -E "[0-9]*[1-9]+ error\(s\), [0-9]+ warning\(s\)" %infile%
copy counterrs.bat d:\scratch\build cd d:\scratch\build\master ..\rebuild ..\counterrs rebuild.out
release.bat
run this script while in your build directory (the one with condor.sln) to build and/or release HTCondor. Optional arguments are
- debug|retail|RelWithDebInfo - set build type, default is retail
- msi - build an MSI rather than releasing into the release_dir
- <dir> - set destination directory for release, default is release_dir
release.bat
@echo off
setlocal
for %%I in (cmake.exe) do if "%%~f$PATH:I"=="" (
echo cmake is not in the path - aborting
exit /b 1
)
if not exist cmake_install.cmake (
echo There is no cmake_install.cmake file in the current directory
echo did you forget to run cmakeit.bat?
exit /b 1
)
set _bldtype_=RelWithDebInfo
if "%1"=="msi" goto msi
if "~%1"=="~" goto doit
if "~%1"=="~." set _install_=-DCMAKE_INSTALL_PREFIX=Release
if NOT "~%1"=="~." set _install_=-DCMAKE_INSTALL_PREFIX=%~sf1
:top
if "~%2"=="~" goto doit
goto %2
:debug
set _bldtype_=Debug
goto next
:relwithdebinfo
:retail
set _bldtype_=RelWithDebInfo
goto next
:msi
set _msi_=condor-pre.msi
goto next
:next
shift
goto top
:doit
if ~%_bldtype_%==~ for /F %%I in ('dir /b all_build.dir') do set _bldtype_=%%I
@echo on
cmake.exe -DBUILD_TYPE=%_bldtype_% %_install_% -P cmake_install.cmake > release.out
@echo off
::@findstr /I /C:"condor_master.exe" release.out
for /F %%I in ('grep -c Installing release.out') do set /A install_count=%%I - 1
for /F %%I in ('grep -c Up-to-date release.out') do set /A uptodate_count=%%I + %install_count%
echo Updated %install_count% of %uptodate_count%
:makemsi
if "%_msi_%"=="" goto :EOF
set _msiname_=
for /f "delims== tokens=3" %%I in ('findstr "condor-" msconfig\wix\xml\condor.xsl') do @set _msiname_=%%~I
if NOT "%_msiname_%"=="" set _msi_=%_msiname_%.msi
:: @echo msi is: "%_msi_%"
if not "%_msi_:~0,6%"=="condor" goto :EOF
:: figure out the release path from the release.out log
for /f "tokens=3" %%I in ('grep condor_master.exe release.out') do @set _master_=%%~dpI
set _reldir_=%_master_:\bin\=%
if not EXIST %_reldir_%\bin\condor_master.exe goto :EOF
@echo makeing msi: %_msi_%
@echo on
%_reldir_%\etc\wix\do_wix %_reldir_% %CD%\%_msi_%
copy release.bat d:\scratch\build cd d:\scratch\build\master ..\release test
build_workspace.bat
This script copies a SHA from the current git repository to submit-3.batlab.org and submits it as a workspace build. This script depends on a shell script in ~johnkn/workspace on submit-3. arguments are:
index | <SHA1>The commit to build, "index" means use the SHA from git write-tree<description>(optional) Description for the workspace buildall | win | win+ | warn(optional), passed on to build_workspace script on submit-3Examplebuild_workspace.bat
@echo off @setlocal if not exist nmi_tools goto usage if "%1"=="" goto usage if "%1"=="-h" goto usage if "%1"=="-help" goto usage if "%1"=="/help" goto usage set sha=%1 shift if "%sha%"=="index" ( for /F %%I in ('git write-tree') do ( set sha=%%I ) ) set submit=submit-3.batlab.org set tar=git archive %sha% set platforms=all set dir=wk%RANDOM% set dir=%dir: =% set arg1=%~1 if NOT "%arg1:~0,1%" == "-" ( set desc=%arg1: =_% set dir=%dir%%arg1:~0,6% shift ) set desc=%desc: =% if NOT "%1"=="" set platforms=%1 @echo on %tar% | plink -2 -i %USERPROFILE%\keys\putty.ppk -l %USERNAME% %submit% cd workspace ; mkdir %dir% ; cd %dir% ; tar xf - ; ../build_workspace %desc% %platforms% @echo off goto :EOF :: :usage @echo This script must be run from the CONDOR_SRC directory @echo Usage: %0 sha1 [description] [all ^| win ^| win+ ^| warn] @echo. send the given sha1 or branch to NMI submit machine as ~/workspace/wk^<rand^> @echo. and then run ~/workspace/build_workspace to build it. @echo. if sha1 is index, then git write-tree is used to generate the sha1copy build_workspace.bat d:\scratch\condor cd d:\scratch\condor\master ..\build_workspace index description all
build_workspace (shell script)
This bash script invokes nmi_condor_submit It should be executed while at the root of a git clone or archive on one of the batlab submit machines. Arguments are optional the first is the description and second is shorthand for combinations of platform. default is to build on Win8 and Fedora21
<description>the description stringallbuild on all platformswinbuild on all windows platformswin+build on windows and fedorawarn|fedorabuild on fedorafastbuild on fedora and solarisdebbuild on all debianExamplebuild_workspace
#!/bin/bash TAG="$1" PLATFORMS="x86_64_Windows8,x86_64_Fedora21" if [[ $2 = "all" || $2 = "ALL" ]] then PLATFORMS="all" fi if [[ $2 = "win" || $2 = "WIN" ]] then PLATFORMS="x86_64_Windows7,x86_64_Windows8" fi if [[ $2 = "win+" || $2 = "WIN+" ]] then PLATFORMS="x86_64_Windows7,x86_64_Windows8,x86_64_Fedora21,x86_64_Fedora22" fi if [[ $2 = "win8" || $2 = "WIN8" ]] then PLATFORMS="x86_64_Windows8" fi if [[ $2 = "fedora" || $2 = "warn" || $2 == "fast" || $2 == "winfast" ]] then PLATFORMS="x86_64_Fedora21,x86_64_Fedora22" if [[ $2 == "fast" || $2 == "winfast" ]] then PLATFORMS="${PLATFORMS},x86_64_Solaris11" fi if [[ $2 == "winfast" ]] then PLATFORMS="${PLATFORMS},x86_64_Windows7,x86_64_Windows8" fi fi if [[ $2 = "f22" ]] then PLATFORMS="x86_64_Fedora22" fi if [[ $2 = "deb" ]] then PLATFORMS="x86_64_Debian6,x86_64_Debian7,x86_64_Debian8" fi pwd if [ -d "nmi_tools" ] then echo "building temporary workspace" pushd nmi_tools else echo "building main workspace" pushd ~johnkn/workspace/CONDOR_SRC/nmi_tools fi pwd ./condor_nmi_submit --build --git --notify-fail-only --workspace=.. --desc="$TAG" -platforms="$PLATFORMS" echo ./condor_nmi_submit --build --git --notify-fail-only --workspace=.. --desc=\"$TAG\" -platforms=\"$PLATFORMS\" pwd popdcp build_workspace ~/workspace cd ~/workspace/wkNNN ../build_workspace
