Download a copy of MPICH2 from the Argonne site:
-* http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.2.1/mpich2-1.2.1-win-ia32.msi
+http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.2.1/mpich2-1.2.1-win-ia32.msi
During the install, make sure to enable use of the tools by all users, and not just yourself.
Once the installation is compleate, ensure that the MPICH pass phrase is the same on all machines in the pool:
@@ -16,7 +16,169 @@
[HKEY_LOCAL_MACHINE\SOFTWARE\MPICH\SMPD]
"phrase"="behappy"
+{endcode}
+
+The final step is to disble the SMPD service. Using the *Microsoft Managment Console*, open the *Services Common Console Document*. Find the entry labelled *MPICH2 Process Manager, Argonne National Lab* and modify it's properties such that it is inactive and will no longer start on boot.
+
+We do this so that Condor can manage the MPI processes itself. That is, we will configure Condor such that it can start, stop and monitor any or all of the MPI related process.
+
+{subsection: Configuration}
+
+The aim of installing MPICH2 is to use it in conjunction with Condor. To this end, we need to make some changes to Condor's configuration, as well as edit some helper scripts.
+
+The following will let Condor manage the =smpd.exe= daemon rather than the Windows Service Manager:
+
+{code}
+##
+# Tell the condor_master what binary to use for MPICH2's
+# comunication daemon.
+##
+SMPD_SERVER = C:\Program Files\MPICH2\bin\smpd.exe
+SMPD_SERVER_ARGS = -p 6666 -d 1
+
+##
+# We are letting Condor spawn and manage the smpd.exe server
+# for us, so we need to tell the condor_master on this machine
+# to spawn a smpd.exe server, in addition to any other daemons
+# it is configured to run.
+##
+DAEMON_LIST = $(DAEMON_LIST), SMPD_SERVER
+{endcode}
+
+Unlike the Condor daemons, we cannot define a log file for the =smpd.exe= daemon in the Condor configuration file, ao we must do this manually:
+
+{code}
+C:\> smpd.exe -set logfile C:\condor\log\SmpdLog
+logfile = C:\condor\log\SmpdLog
+{endcode}
+
+Note that we only need to do this once, since the option will be saved persistently in the registry.
+
+We use the following configuration on the dedicated pool machines that will be used for MPI jobs:
+
+{code}
+DAEMON_LIST = MASTER, STARTD
+
+WANT_SUSPEND = False
+WANT_VACATE = False
+START = True
+SUSPEND = False
+CONTINUE = True
+PREEMPT = False
+KILL = False
+RANK = 0
+
+DedicatedScheduler = "DedicatedScheduler@real.host.name"
+STARTD_EXPRS = $(STARTD_EXPRS), DedicatedScheduler
+{endcode}
+
+This configuration will allow Condor to always runs jobs.
+
+{subsection:Helper Script}
+
+Once Condor has been configured to run parallel jobs, we must create a helper script such that Condor can use it to interact with the new MPICH2 installation.
+
+The following can be used as a base for a more complex driver:
+
+{code}
+set _CONDOR_PROCNO=%_CONDOR_PROCNO%
+set _CONDOR_NPROCS=%_CONDOR_NPROCS%
+
+REM If not the head node, just sleep forever
+if not [%_CONDOR_PROCNO%] == [0] copy con nothing
+
+REM Set this to the bin directory of MPICH installation
+set MPDIR="C:\Program Files\MPICH2\bin"
+
+REM run the actual mpijob
+%MPDIR%\mpiexec.exe -n %_CONDOR_NPROCS% -p 6666 %*
+
+exit 0
+{endcode}
+{subsection:Submitting a an MPI Job}
+
+For our purposes we will use the sample shipped with the Windows version of MPICH2 to demostrate how to use Condor and MPI on Windows:
+
+{code}
+C:\demo> cp "C:\Program Files\MPICH2\examples\cpi.exe" .
+ 1 file(s) copied.
+{endcode}
+
+We also need to create an input file:
+
+{code}
+10
+100
+1000
+10000000000000
+0
+{endcode}
+
+Where the last input, =0=, terminates the =cpi.exe= application.
+
+All that remains is to define a submit file:
+
+{code}
+universe = parallel
+executable = mp2script.bat
+arguments = cpi.exe
+machine_count = 1
+input = input.file
+output = out.$(NODE).log
+error = error.$(NODE).log
+log = work.$(NODE).log
+should_transfer_files = yes
+when_to_transfer_output = on_exit
+transfer_input_files = cpi.exe
+queue
+{endcode}
+
+Where =mp2script.bat= is the helper script, and =input.file= is the input file.
+
+Now we can submit our MPI application to Condor:
+
+{code}
+C:\demo> condor_submit parallel.submit
+Submitting job(s).
+Logging submit event(s).
+1 job(s) submitted to cluster 18.
+
+...
+{endcode}
+
+{code}
+C:\demo> condor_q
+-- Submitter: marge : <172.16.46.137:4301> : marge
+ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
+ 18.0 Administrator 2/17 16:10 0+00:00:11 R 0 0.0 mp2script.bat cpi.
+
+1 jobs; 0 idle, 1 running, 0 held
+
+...
+{endcode}
+
+{code}
+C:\demo> condor_q
+-- Submitter: marge : <172.16.46.137:4301> : marge
+ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
+
+0 jobs; 0 idle, 0 running, 0 held
+{endcode}
+
+Look at =out.log= to see the result of the run:
+
+{code}
+C:\demo> more out.log
+Enter the number of intervals: (0 quits) pi is approximately 3.1424259850010987, \
+Error is 0.0008333314113056 wall clock time = 0.000014
+Enter the number of intervals: (0 quits) pi is approximately 3.1416009869231254, \
+Error is 0.0000083333333323 wall clock time = 0.000007
+Enter the number of intervals: (0 quits) pi is approximately 3.1415927369231227, \
+Error is 0.0000000833333296 wall clock time = 0.000637
+Enter the number of intervals: (0 quits) pi is approximately 3.1415926535898295, \
+Error is 0.0000000000000364 wall clock time = 12.460265
+Enter the number of intervals: (0 quits)
{endcode}
-The final step is to disble the SMPD service. Using the Microsoft Managment Console, open the Services Common Console Document. Find the entry labled "MPICH2 Process Manager, Argonne National Lab" and modify it's properties such that it is not running and it's start-up is disabled.
+Where the output are is result of an inputs of =10=, =100=, =1000=, =10000000000000=, and =0= respectively.
