Page History

Turn Off History

Condor Qpid Messaging System

The Condor Qpid Messaging System can be divided into 3 units.

  1. Message Publisher (client producer)
  2. broker where the messaging Queues are initialized (server)
  3. Message Listener (client consumer)

At a higher level the interaction between these layers is depicted in the following diagram.

qpid.jpeg Figure 1

The Message Publisher sends messages to the queue in the borker. This is similar to a PUT Http request where the client sends new content to the the server. The message thus obtained from the Message Publisher is appended to the target queue. The messages in the queue are maintain persistently.When a Message Listener establishes connection with the queue and requests for messages,they are forwarded to the listener. Again this interaction is similar to a GET Http request.

A detailed description of each of these three units follows.

Qpid Message Publisher:

Event logs for jobs submitted with Condor are available at the SchedD. These log files are rotated over to maintain the latest 'X' event logs. A system that would read these event logs and persistently forward these logs is the core functionality of the Message Publisher.

The event log forwarder makes use of a log reader api(written by Nick).The forwarder works as a wrapper on top of the log reader api, providing control to selectively include or exclude event types of the job whose logs are to be read and forwarded.

The challenges faced by this system are

  1. avoid loss of event logs.
  2. avoid duplicate log messages
  3. message transformation

The above stated challenges are at the forwarder part assuming the event log reader api is consistent with these challenges too. Thus after the event logs(event type(s) can be specified) are read from the log files, it is formatted. Currently the log message can be modified to the syslogng format or the classAd format or simply made as a sequence of characters (with or without separators). The abstraction on the formatting api provides easy scope to plugin other formats.

Thus the log events can be converted to messages of required format and then forwarded.

Again forwarding could mean simply to write into another text file(locally/remotely) or send messages via a syslogng or to a messaging queue system. In this documentation we are to discuss about the interactions with a messaging system.

This log forwarder is then made into a Qpid client that publishes the logs read to queues in a messaging broker.

In figure 2 the various components of the Event Log Forwarder and integration with Qpid as a client publisher is given.

condorQpidInt.jpeg Figure 2

Figure 3 provides the flow chart explanation of the operational logic.

condorQpidFlow.jpeg Figure 3

SOURCE CODE:

This Message Publisher client code is run as a condor job.

Qpid Messaging System:

Apache's Qpid Messaging System plays the middle man role. Given below is some notes about Qpid.

Enterprise Messaging systems let programs communicate by exchanging messages, much as people communicate by exchanging email. Unlike email, enterprise messaging systems provide guaranteed delivery, speed, security, and freedom from spam. Until recently, there was no open standard for Enterprise Messaging systems, so programmers either wrote their own, or used expensive proprietary systems.

AMQP Advanced Message Queuing Protocol is the first open standard for Enterprise Messaging. It is designed to support messaging for just about any distributed or business application. Routing can be configured flexibly, easily supporting common messaging paradigms like point-to-point, fanout, publish-subscribe, and request-response.

Apache Qpid implements the latest AMQP specification, providing transaction management, queuing, distribution, security, management, clustering, federation and heterogeneous multi-platform support and a lot more. And Apache Qpid is extremely fast.

More information about Qpid and its implentation,configurational and setup details can be found at [http://qpid.apache.org/]

The Qpid broker is configured to use the publisher-subscriber routing model and store messages persistently. Also the queues in the broker and their corresponding routing keys are used to initialize queues in the Qpid broker.

SOURCE CODE:

Qpid Message Listener:

This qpid client is used for requesting messages from the broker queues. Multiple message listeners can run concurrently each tied to a different or same queue to read messages. In our implementation we have a listener that can either write the listened log messages to a file(local/remote) after formatting or send in emails or send instance messages.

SOURCE CODE:

Figure 4 details the integration of the above explained 3 components:

condorQpidInt2.jpeg Figure 4

CONDOR ADD-ON:

The integration of the event log forwarder with the messaging system is added into the condor source code as a daemon ,condor job implementation. condor_qpid daemon:

The qpid broker is run as a condor daemon – condor_qpid that is controlled by the condor_master. It performs the following functions:

  1. Declare queues and bind them to the exchange
  2. run the qpid broker on a free available port that is dynamically chosen
  3. publish a class Ad containing meta details about the qpid broker such as port #, host name, queue name, routing key

condor jobs:

The qpid clients that send messages to the broker queues as well and listen/retrieve messages run as condor jobs that poll for the condor_qpid daemon published classAd for meta details needed to connect to the broker queues.

Figure 5 explains the control flow between the condor qpid broker daemon and client condor jobs.

condorQpidDaemon.jpeg Figure 5

Attachments: