{section: How to configure MPI on Windows}

Known to work with HTCondor version: 7.5

_Note: The good folks at Kitware have also posted a {link: http://www.itk.org/Wiki/Proposals:Condor#MPICH2_on_Windows comprehensive webpage detailing their setup using mpich2 with HTCondor on Windows}_

{subsection:Installation}

Download a copy of MPICH2 from the Argonne site:

http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.2.1/mpich2-1.2.1-win-ia32.msi

During the install, make sure to enable use of the tools by all users, and not just yourself.
Once the installation is compleate, ensure that the MPICH pass phrase is the same on all machines in the pool:

{code}
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\MPICH\SMPD]
"phrase"="behappy"
{endcode}

The final step is to disble the SMPD service.  Using the *Microsoft Managment Console*, open the *Services Common Console Document*.  Find the entry labelled *MPICH2 Process Manager, Argonne National Lab* and modify it's properties such that it is inactive and will no longer start on boot.

We do this so that HTCondor can manage the MPI processes itself.  That is, we will configure HTCondor such that it can start, stop and monitor any or all of the MPI related process.

{subsection: Configuration}

The aim of installing MPICH2 is to use it in conjunction with HTCondor.  To this end, we need to make some changes to HTCondor's configuration, as well as edit some helper scripts.

The following will let HTCondor manage the =smpd.exe= daemon rather than the Windows Service Manager:

{code}
##
# Tell the condor_master what binary to use for MPICH2's
# comunication daemon.
##
SMPD_SERVER = C:\Program Files\MPICH2\bin\smpd.exe
SMPD_SERVER_ARGS = -p 6666 -d 1

##
# We are letting HTCondor spawn and manage the smpd.exe server
# for us, so we need to tell the condor_master on this machine
# to spawn a smpd.exe server, in addition to any other daemons
# it is configured to run.
##
DAEMON_LIST = $(DAEMON_LIST), SMPD_SERVER
{endcode}

Unlike the HTCondor daemons, we cannot define a log file for the =smpd.exe= daemon in the HTCondor configuration file, ao we must do this manually:

{code}
C:\> smpd.exe -set logfile C:\condor\log\SmpdLog
logfile = C:\condor\log\SmpdLog
{endcode}

Note that we only need to do this once, since the option will be saved persistently in the registry.

We use the following configuration on the dedicated pool machines that will be used for MPI jobs:

{code}
DAEMON_LIST 	= MASTER, STARTD

WANT_SUSPEND	= False
WANT_VACATE	= False
START		= True
SUSPEND		= False
CONTINUE	= True
PREEMPT		= False
KILL		= False
RANK		= 0

DedicatedScheduler = "DedicatedScheduler@real.host.name"
STARTD_EXPRS 	= $(STARTD_EXPRS), DedicatedScheduler
{endcode}

This configuration will allow HTCondor to always runs jobs.

{subsection:Helper Script}

Once HTCondor has been configured to run parallel jobs, we must create a helper script such that HTCondor can use it to interact with the new MPICH2 installation.

The following can be used as a base for a more complex driver:

{code}
set _CONDOR_PROCNO=%_CONDOR_PROCNO%
set _CONDOR_NPROCS=%_CONDOR_NPROCS%

REM If not the head node, just sleep forever
if not [%_CONDOR_PROCNO%] == [0] copy con nothing

REM Set this to the bin directory of MPICH installation
set MPDIR="C:\Program Files\MPICH2\bin"

REM run the actual mpijob
%MPDIR%\mpiexec.exe -n %_CONDOR_NPROCS% -p 6666 %*

exit 0
{endcode}

{subsection:Submitting a an MPI Job}

For our purposes we will use the sample shipped with the Windows version of MPICH2 to demostrate how to use HTCondor and MPI on Windows:

{code}
C:\demo> cp "C:\Program Files\MPICH2\examples\cpi.exe" .
       1 file(s) copied.
{endcode}

We also need to create an input file:

{code}
10
100
1000
10000000000000
0
{endcode}

Where the last input, =0=, terminates the =cpi.exe= application.

All that remains is to define a submit file:

{code}
universe = parallel
executable = mp2script.bat
arguments = cpi.exe
machine_count = 1
input = input.file
output = out.$(NODE).log
error = error.$(NODE).log
log = work.$(NODE).log
should_transfer_files = yes
when_to_transfer_output = on_exit
transfer_input_files = cpi.exe
queue
{endcode}

Where =mp2script.bat= is the helper script, and =input.file= is the input file.

Now we can submit our MPI application to HTCondor:

{code}
C:\demo> condor_submit parallel.submit
Submitting job(s).
Logging submit event(s).
1 job(s) submitted to cluster 18.

...
{endcode}

{code}
C:\demo> condor_q
-- Submitter: marge : <172.16.46.137:4301> : marge
ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
  18.0   Administrator   2/17 16:10   0+00:00:11 R  0   0.0  mp2script.bat cpi.

1 jobs; 0 idle, 1 running, 0 held

...
{endcode}

{code}
C:\demo> condor_q
-- Submitter: marge : <172.16.46.137:4301> : marge
ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD

0 jobs; 0 idle, 0 running, 0 held
{endcode}

Look at =out.log= to see the result of the run:

{code}
C:\demo> more out.log
Enter the number of intervals: (0 quits) pi is approximately 3.1424259850010987, \
Error is 0.0008333314113056 wall clock time = 0.000014
Enter the number of intervals: (0 quits) pi is approximately 3.1416009869231254, \
Error is 0.0000083333333323 wall clock time = 0.000007
Enter the number of intervals: (0 quits) pi is approximately 3.1415927369231227, \
Error is 0.0000000833333296 wall clock time = 0.000637
Enter the number of intervals: (0 quits) pi is approximately 3.1415926535898295, \
Error is 0.0000000000000364 wall clock time = 12.460265
Enter the number of intervals: (0 quits)
{endcode}

Where the output are is result of an inputs of =10=, =100=, =1000=, =10000000000000=, and =0= respectively.