{section: How to configure multiple negotiators}

Known to work in HTCondor version: 8.5.4

This is a technique for increasing the scalability of the HTCondor negotiator. In very large pools (200,000 slots), the matchmaking process can take a long time. This can become a problem for glidein pools, where it's important for an unclaimed machine to be matched quickly (glideins will often exit if unclaimed for more than 10 minutes).

The idea is to divide the machine ads between multiple negotiator daemons, which can do matchmaking in parallel. The negotiators communicate with all schedds in the pool, but consider only a subset of the machine ads.

The division of machines is by defining an expression for each negotiator that evaluates to true for machines it should manage. You need to write these expressions carefully so that all machines will match to one and only one negotiator.

One drawback to this configuration is that each negotiator does its own accounting, so fair-share and quotas operate independently for each portion of the pool.

{code}
# Setup alternate negotiators so we can partition the pool.
# We run one negotiator for T2_US_*, one for T1_*/T2_CH_CERN, and one for everyone else.
# Right now, this is as close as we have to splitting the pool into thirds (the "everyone else"
# negotiator is still a bit bigger).
#
# Define the new negotiator daemons; add them to the daemon list:
NEGOTIATORT1 = $(NEGOTIATOR)
NEGOTIATORT1_ARGS = -f -local-name NEGOTIATORT1
NEGOTIATOR.NEGOTIATORT1.NEGOTIATOR_LOG = $(LOG)/NegotiatorT1Log
NEGOTIATOR.NEGOTIATORT1.MATCH_LOG = $(LOG)/MatchT1Log
NEGOTIATOR.NEGOTIATORT1.SPOOL = $(SPOOL)/negotiator_t1

NEGOTIATORUS = $(NEGOTIATOR)
NEGOTIATORUS_ARGS = -f -local-name NEGOTIATORUS
NEGOTIATOR.NEGOTIATORUS.NEGOTIATOR_LOG = $(LOG)/NegotiatorUSLog
NEGOTIATOR.NEGOTIATORUS.MATCH_LOG = $(LOG)/MatchUSLog
NEGOTIATOR.NEGOTIATORUS.SPOOL = $(SPOOL)/negotiator_us

VALID_SPOOL_FILES=$(VALID_SPOOL_FILES), negotiator_us, negotiator_t1
DAEMON_LIST = $(DAEMON_LIST), NEGOTIATORT1, NEGOTIATORUS
DC_DAEMON_LIST =+ NEGOTIATORT1, NEGOTIATORUS

# Make sure each negotiator considers the right set of slots.
NEGOTIATOR_SLOT_CONSTRAINT=!regexp("^(T1_|T2_US_|T2_CH_CERN)", GLIDEIN_CMSSite)
NEGOTIATOR.NEGOTIATORT1.NEGOTIATOR_SLOT_CONSTRAINT=regexp("^(T1_|T2_CH_CERN)", GLIDEIN_CMSSite)
NEGOTIATOR.NEGOTIATORUS.NEGOTIATOR_SLOT_CONSTRAINT=regexp("^T2_US_", GLIDEIN_CMSSite)
{endcode}

If using HAD, you also need to set up multiple HAD and Replication daemons.

{code}
HADT1 = $(HAD)
HADT1_ARGS = -f -local-name HADT1 -p 51451
HADT1_ENVIRONMENT = "_CONDOR_HAD_PORT=51451 _CONDOR_REPLICATION_PORT=41451"
HAD.HADT1.HAD_LOG = $(LOG)/HADT1Log
HAD.HADT1.HAD_CONTROLLEE = NEGOTIATORT1
MASTER_NEGOTIATORT1_CONTROLLER = HADT1

HADUS = $(HAD)
HADUS_ARGS = -f -local-name HADUS -p 51452
HADUS_ENVIRONMENT = "_CONDOR_HAD_PORT=51452 _CONDOR_REPLICATION_PORT=41452"
HAD.HADUS.HAD_LOG = $(LOG)/HADUSLog
HAD.HADUS.HAD_CONTROLLEE = NEGOTIATORUS
MASTER_NEGOTIATORUS_CONTROLLER = HADUS

REPLICATIONT1 = $(REPLICATION)
REPLICATIONT1_ARGS = -f -local-name REPLICATIONT1 -p 41451
REPLICATIONT1_ENVIRONMENT = "_CONDOR_REPLICATION_PORT=41451 _CONDOR_SPOOL=$(SPOOL)/negotiator_t1"
REPLICATION.REPLICATIONT1.REPLICATION_LOG = $(LOG)/ReplicationT1Log

REPLICATIONUS = $(REPLICATION)
REPLICATIONUS_ARGS = -f -local-name REPLICATIONUS -p 41452
REPLICATIONUS_ENVIRONMENT = "_CONDOR_REPLICATION_PORT=41452 _CONDOR_SPOOL=$(SPOOL)/negotiator_us"
REPLICATION.REPLICATIONUS.REPLICATION_LOG = $(LOG)/ReplicationUSLog

DC_DAEMON_LIST =+ HADT1, HADUS, REPLICATIONT1, REPLICATIONUS
DAEMON_LIST = $(DAEMON_LIST), HADT1, HADUS, REPLICATIONT1, REPLICATIONUS
{endcode}