Recently in AMQP Category

My colleague Matt (spinningmatt) posted a really useful article, Submitting Jobs with AMQP on a Condor Based Grid. The article includes an example C++ program that uses a low-latency job submission feature of MRG.

The low-latency feature uses AMQP (MRG's Messaging) to deliver job workloads to a grid's execution nodes.  This bypasses the job scheduler (Condor's schedd).  Instead a special daemon on the execute node consume jobs off an AMQP queue.  It's a pull versus push model.

Why is this useful?  Well originally the intent was for sub-second jobs in the financial services industry.  Consider a grid of index calculation applications. Each calculation may take less than a second to perform.  The applications can't wait for a job scheduler to decide that their execution nodes are available, then schedule a job onto the node, then push the job out there.  Instead jobs are placed on AMQP queues.  As soon as an execute node is free to perform work it pulls the next job form the queue. There is barely any latency between jobs.

Of course as soon as you do this for one specific use, sub-second jobs, then others see the advantages too. This feature doesn't need to be sub-second jobs. MRG customers from various industries now see the advantage of this feature.

There are some considerations. For example, should the entire grid be untilized this way or should a specific portion of the grid be carved off for low-latency workloads?  If the jobs are sub-second and high volume, should they be reported to a management console (this could cause quite a bit of clutter) or just logged? How should failed jobs be managed under different scenarios? e.g. in sub-second transactions a failed job may have missed its window of being useful and therefore there is little point in resubmitting. The answers to these quesitons will depend on the type of low-latency workload.

If you are interested in more information on this topic please remember to check out Matt's post.

So over the weekend I posted an entry that introduced some of the AMQP benefits. That article was very upbeat on AMQP and Qpid.  Two areas of improvement are:

• Potential complexity of feature configuration
• Lack of concrete or industry specific examples
  

Improvement in the second area can really help solve the first issue.

As standards mature they can often get too complex. I remember when CORBA 2.2 came out and I read about Portable Object Adapters.  I was impressed at the flexibility it provided but I was also anxious that the number of configuration options would overwhelm and discourage many developers.  So while it's desirable to have lots of enhanced features, you also don't want lots and lots of features, and their combinations of configurations, to put off the novice.

Of course much of the complexity issue can be solved by providing good examples. However, examples of how to code a feature don't always give you the context. It is better if we can explain the feature with a real world example. When I developed the current C++ training examples for MRG Messaging we (at Red Hat) decided to try and build examples around a banking use case.  And recently I had an example called "tradedemo" committed to the Qpid project.

This week I saw some example code I submitted to the Apache/Qpid project committed. The example demonstrates a possible use case for some Qpid queue features.  I suspect we're going to see more examples with use case focus.  That is a very good thing and I'd encourage Qpid developers to take time to submit such examples.  We need a richer set. I'll discuss this and specifically my example on a later post.

This post is about AMQP and Qpid and is long overdue.

Look, no doubt I'm biased but the more and more I look at AMQP, the more I'm amazed with it's flexibility and it's depth and breadth of features. Its importance in the financial services industry is growing and its profile is rising in other industries (animation, energy, logistics, telecommunications etc.) and government.  

What makes AMQP so important? Well there are three basic characteristics that make it compelling :

• Standard wire protocol
• Multi-platform, multi-language support
• The model - simple, yet flexible
  

Standard wire protocol opens up the ecosystem and provides interoperability and freedom from vendor lock-in. We saw this with IIOP in the CORBA specification. Frankly I'm surprised and disappointed that it has taken this long to get a standard wire protocol for "messaging". I have messaging in quotes because no doubt that there are many that will point to various standards like IIOP, and HTTP and SOAP etc.  Those of you that ave been working on enterprise systems with a more asynchronous messaging approach know what I mean.

Because of the standard wire-protocol it means that anyone can write on any platform or language and still take advantage of AMQP. The Qpid project currently supports C++, Java (including JMS), Python, Ruby, .NET (C#).  In fact Microsoft have joined AMQP and joined Qpid! Hardware vendors like Cisco participate. The point is the ecosystem is growing.

This is very ambitious but as all these various languages and platforms with their various characteristics take advantage of AMQP? Is routing text messages through telecommunications exchanges, large image files to render farms, and low-latency high volume market data a reasonable requirement to be demanded off of one standard? That's where the flexibility comes in.  Unlike other messaging systems that might focus just on a single language programming API, or on a single technology like multicast, or on a single abstraction like a queue, AMQP's model is very flexible.  And yet it remains quite simple.


AMQP has three main abstractions:

• An exchange - where messages are posted
• A queue - where consumers find messages
• A binding - that binds a specific message to one or more queues

The decoupling of message sending from message delivery makes this model very powerful.  Rather than specifying a specific queue or forcing a broadcast across a network, AMQP provides a much more flexible model.

There is a nice explanation of these concepts here as part of the MRG documentation.

And so that is one of the reasons I joined Red Hat last May. AMQP and Qpid represent a new wave in enterprise application communication and I'm very excited in the interest and growth of adoption I'm seeing so far.  This along with the Condor based Grid technology is a very powerful distributed computing platform for government and industry.  It many ways it's what many of use were thinking would happen well over a decade ago.  The growth of higher speed networks and the ubiquity of messaging in every facet of our lives are pushing technology like this to broader adoption.  In the 21st century we are going to need a much more standard and ubiquitous messaging infrastructure.

Tomorrow I'll post about the example I committed to Qpid.