HTCondorWiki: Igor Stress Test

Page History

We have found it useful to stress-test a release candidate by giving it to Igor Sfiligoi to run in glideinWMS. This exercises a lot of Condor features that are important to OSG-type users: glidein, Condor-G, CE schedd, CCB. Igor suggested that it would be a good idea of we could run this test in-house, and we agree, so we have set up the machinery to do so. This document describes roughly how we installed it.

High-level View

ghost-pool - runs light-weight jobs on cluster nodes. The setup is in /unsup/condor/ghost-pool. Some nodes in CHTC also join the ghost pool. These are configured in the CHTC cfengine.
CE: vdt-itb.cs.wisc.edu - an OSG CE that is part of the ITB (integration test bed). This node will be referred to as the CE. The schedd on this node is part of the ghost pool. The jobmanager is vdt-itb.cs.wisc.edu/jobmanager-nfslitecondor.
factory: c157.cs.wisc.edu - the glideinWMS factory, installed in /data1/itb-stress. It submits glideins to the CE. This node is part of the GLOW pool (despite its domain name). It normally runs condor jobs. Turn off condor when doing a large-scale test.
submit: c158.cs.wisc.edu - the glideinWMS VO frontend, user job submission node, and collector tree for the glidein pool, all installed in /data1/itb-stress. This node normally runs condor jobs. Turn off condor when doing a large-scale test.

glidein factory setup

Go to c157.chtc.wisc.edu. Install the factory in /data1/itb-stress. Where an unprivileged user account is needed, we are using zmiller.

Documentation for the glideinWMS factory may be found here: http://www.uscms.org/SoftwareComputing/Grid/WMS/glideinWMS/doc.v2/install/factory_install.html

cd /data1/itb-stress
cvs -d :pserver:anonymous@cdcvs.fnal.gov:/cvs/cd_read_only co -r snapshot_091214 glideinWMS

perl-Time-HiRes appears to be already installed as part of the main perl install

httpd was already installed, so just enabled it:
/sbin/chkconfig --level 3  httpd on
/etc/init.d/httpd start

wget http://atlas.bu.edu/~youssef/pacman/sample_cache/tarballs/pacman-3.28.tar.gz
tar xvzf pacman-3.28.tar.gz
cd pacman-3.28
. ./setup.sh
cd ..

mkdir osg_1_2_4
cd osg_1_2_4/

export VDTSETUP_CONDOR_CONFIG=/data1/itb-stress/personal-condor/etc/condor_config
export VDTSETUP_CONDOR_BIN=/data1/itb-stress/personal-condor/bin

pacman -get http://software.grid.iu.edu/osg-1.2:client
. ./setup.sh

ln -s /etc/grid-security/certificates globus/TRUSTED_CA

cd ..

yum install rrdtool-python


M2Crypto python module appears to be already installed
  m2crypto-0.16-6.el5.3.x86_64
  This is older than Igor's requirements 0.17, but I am trying it to see if it works.

Download javascriptrrd-0.4.2.zip and unzip it.

Download flot-0.5.tar.gz and untar it.

I made a service cert for glideinwms and installed it in certs.  (I used the following command.)
cert-gridadmin -email dan@hep.wisc.edu -affiliation osg -vo glow -host c157.chtc.wisc.edu -service glideinwms

A proxy for the factory may be made with this command:
grid-proxy-init -cert certs/c157.chtc.wisc.edu_glideinwms_cert.pem -key certs/c157.chtc.wisc.edu_glideinwms_key.pem -hours 100 -out certs/c157.chtc.wisc.edu_proxy.pem


cd glideinWMS/install
chown zmiller /data1/itb-stress
su zmiller
./glideinWMS_install


choose [4] pool Collector
Where do you have the Condor tarball? /afs/cs.wisc.edu/p/condor/public/binaries/v7.4/7.4.1/condor-7.4.1-linux-x86_64-rhel5-dynamic.tar.gz

Where do you want to install it?: [/home/zmiller/glidecondor] /data1/itb-stress/glidecondor
Where are the trusted CAs installed?: [/data1/itb-stress/osg_1_2_4/globus/TRUSTED_CA] /etc/grid-security/certificates


Will you be using a proxy or a cert? (proxy/cert) cert
Where is your certificate located?: /data1/itb-stress/certs/c157.chtc.wisc.edu_glideinwms_cert.pem
Where is your certificate key located?: /data1/itb-stress/certs/c157.chtc.wisc.edu_glideinwms_key.pem


What name would you like to use for this pool?: [My pool] ITB-Stress
How many secondary schedds do you want?: [9]

In this version of glideinWMS, the condor_config.local needs to be edited
to replace HOSTALLOW with ALLOW.


./glideinWMS_install
[2] Glidein Factory

Where is your proxy located?: /data1/itb-stress/certs/c157.chtc.wisc.edu_proxy.pem
Where will you host your config and log files?: [/home/zmiller/glideinsubmit] /data1/itb-stress/glideinsubmit

Give a name to this Glidein Factory?: [mySites-c157] ITB-Stress

Do you want to use CCB (requires Condor 7.3.0 or better)?: (y/n) y
Do you want to use gLExec?: (y/n) n
Force VO frontend to provide its own proxy?: (y/n) [y]
Do you want to fetch entries from RESS?: (y/n) [n]
Do you want to fetch entries from BDII?: (y/n) [n]

Entry name (leave empty when finished): ITB
Gatekeeper for 'ITB': vdt-itb.cs.wisc.edu/jobmanager-condornfslite
RSL for 'ITB':
Work dir for 'ITB':
Site name for 'ITB': [ITB]

Should glideins use the more efficient Match authentication (works for Condor v7.1.3 and later)?: (y/n) y
Do you want to create the glidein (as opposed to just the config file)?: (y/n) [n]

/data1/itb-stress/glideinWMS/creation/create_glidein /data1/itb-stress/glideinsubmit/glidein_v1_0.cfg/glideinWMS.xml

Submit files can be found in /data1/itb-stress/glideinsubmit/glidein_v1_0
Support files are in /var/www/html/glidefactory/stage/glidein_v1_0
Monitoring files are in /var/www/html/glidefactory/monitor/glidein_v1_0

cd /data1/itb-stress
# make setup.sh set X509_CERT_DIR and X509_USER_PROXY
. ./setup.sh
glideinsubmit/glidein_v1_0/factory_startup start


I added the following entries to glidecondor/certs/condor_mapfile:
GSI "/DC=org/DC=doegrids/OU=Services/CN=glideinwms/c158.chtc.wisc.edu" vofrontend
FS zmiller gfactory

The FS mapping of zmiller to gfactory appears to be necessary to make the factory
ad have AuthenticatedIdentity gfactory, which is what I configured the VO frontend
to expect.  Not sure why it is using FS to authenticate to the collector though.