So over the weekend I posted an entry that introduced some of the AMQP benefits. That article was very upbeat on AMQP and Qpid. Two areas of improvement are:
As standards mature they can often get too complex. I remember when CORBA 2.2 came out and I read about Portable Object Adapters. I was impressed at the flexibility it provided but I was also anxious that the number of configuration options would overwhelm and discourage many developers. So while it's desirable to have lots of enhanced features, you also don't want lots and lots of features, and their combinations of configurations, to put off the novice.
Of course much of the complexity issue can be solved by providing good examples. However, examples of how to code a feature don't always give you the context. It is better if we can explain the feature with a real world example. When I developed the current C++ training examples for MRG Messaging we (at Red Hat) decided to try and build examples around a banking use case. And recently I had an example called "tradedemo" committed to the Qpid project.
- Potential complexity of feature configuration
- Lack of concrete or industry specific examples
As standards mature they can often get too complex. I remember when CORBA 2.2 came out and I read about Portable Object Adapters. I was impressed at the flexibility it provided but I was also anxious that the number of configuration options would overwhelm and discourage many developers. So while it's desirable to have lots of enhanced features, you also don't want lots and lots of features, and their combinations of configurations, to put off the novice.
Of course much of the complexity issue can be solved by providing good examples. However, examples of how to code a feature don't always give you the context. It is better if we can explain the feature with a real world example. When I developed the current C++ training examples for MRG Messaging we (at Red Hat) decided to try and build examples around a banking use case. And recently I had an example called "tradedemo" committed to the Qpid project.
The example focuses on two features:
TTL is essentially about putting a timeout on a message. The current operation is on a specific message. i.e. each time you want this feature you must set it on the message being published. Time-To-Live is used in other technologies like IP. In IP, a packet may have a TTL associated with the number of hops on the network. But like IP, AMQP associates the TTL on the message (packet). (More on this later.)
A use case for this might be where several consumers are subscribed to the same queue (shared) and each wants to consume all messages. E.g. if several consumers were subscribed for specific stock quotes. Because none of the consumers are actually acquiring the message off of the queue, the message will remain on the queue. This is not desirable because the queue will fill up. Putting a TTL on the message means that it will be removed after a certain period of time. But how long? Well that depends. New subscribers might want to get some history of quotes for that stock. However obviously anything "too old" is of no interest. So perhaps a TTL of 3-5 seconds makes sense for this use case.
This brings up another issue. One can imagine that unlike IP, AMQP messages can be consumed by different types of user. Putting a TTL on the message itself might not make sense. E.g. one consumer might want to subscribe and receive all quotes for a stock for the entire day to examine trends etc. However another type of use may just be interested in the most recent quotes (i.e. within 2 seconds). Therefore it is worth considering placing TTL on a queue rather than a message. This has not being implemented yet in Qpid and I don't believe it is part of the AMQP specification.
So, to summarize, why use TTL? Well for messages that either won't be delivered for some reason or that will be shared and therefore browsed and not consumed. Both of these mean the message won't be acquired (in AMQP terms) - one by design the other by exception.
LVQ has a similar yet different use case. Last Value Queue is exactly that, it contains the last message for a particular routing key.
A use case might be market data. A consumer might not be interested in previous volume, high low prices, and market cap. figures. The history is relevant to that consumer. Rather the consumer just wants the last value. So if market data for a particular symbol (key) is updated frequently the consumer only wants the latest values. (So though TTL sounds like it could achieve this, LVQ is much better.)
In this example I'm assuming an exclusive queue per consumer. Shared queues could also be used. The QueueOptions is what allows you to setup the queue as an LVQ.
You can find the tradedemo example in the C+ examples directory. Download using svn:
Some other Queue Tips
Remember that if you are setting up a shared queue you ought to do this separately like I have done in this example in a declare_queues.cpp. You could also use a python script.
If you need an exclusive queue for the consumer, then make sure you give it a unique name. The Qpid examples do this by adding the session ID to make it unique. So the example above actually sets the queue name used in the declare and the binding as:
UPDATE
You can also browse the code for the tradedemo example here.
- Time-To-Live (TTL)
- Last Value Queue (LVQ)
TTL is essentially about putting a timeout on a message. The current operation is on a specific message. i.e. each time you want this feature you must set it on the message being published. Time-To-Live is used in other technologies like IP. In IP, a packet may have a TTL associated with the number of hops on the network. But like IP, AMQP associates the TTL on the message (packet). (More on this later.)
A use case for this might be where several consumers are subscribed to the same queue (shared) and each wants to consume all messages. E.g. if several consumers were subscribed for specific stock quotes. Because none of the consumers are actually acquiring the message off of the queue, the message will remain on the queue. This is not desirable because the queue will fill up. Putting a TTL on the message means that it will be removed after a certain period of time. But how long? Well that depends. New subscribers might want to get some history of quotes for that stock. However obviously anything "too old" is of no interest. So perhaps a TTL of 3-5 seconds makes sense for this use case.
unsigned int ttl_time = 4000; // 4 seconds
message.getDeliveryProperties().setTtl(ttl_time);
This brings up another issue. One can imagine that unlike IP, AMQP messages can be consumed by different types of user. Putting a TTL on the message itself might not make sense. E.g. one consumer might want to subscribe and receive all quotes for a stock for the entire day to examine trends etc. However another type of use may just be interested in the most recent quotes (i.e. within 2 seconds). Therefore it is worth considering placing TTL on a queue rather than a message. This has not being implemented yet in Qpid and I don't believe it is part of the AMQP specification.
So, to summarize, why use TTL? Well for messages that either won't be delivered for some reason or that will be shared and therefore browsed and not consumed. Both of these mean the message won't be acquired (in AMQP terms) - one by design the other by exception.
LVQ has a similar yet different use case. Last Value Queue is exactly that, it contains the last message for a particular routing key.
A use case might be market data. A consumer might not be interested in previous volume, high low prices, and market cap. figures. The history is relevant to that consumer. Rather the consumer just wants the last value. So if market data for a particular symbol (key) is updated frequently the consumer only wants the latest values. (So though TTL sounds like it could achieve this, LVQ is much better.)
QueueOptions qo; qo.setOrdering(LVQ); std::string binding = "MRKT" + ".#"; session.queueDeclare(arg::queue=queue,
arg::exclusive=true,
arg::arguments=qo); session.exchangeBind(arg::exchange="amq.topic",
arg::queue=queue,
arg::bindingKey=binding);
In this example I'm assuming an exclusive queue per consumer. Shared queues could also be used. The QueueOptions is what allows you to setup the queue as an LVQ.
You can find the tradedemo example in the C+ examples directory. Download using svn:
svn co https://svn.apache.org/repos/asf/qpid/trunk/qpid/cpp/examples/tradedemo .There is another example of using LVQ on the Qpid site here. I am going to include a description of this demo on the Qpid site once it has been approved. I'll post when that is up there.
Some other Queue Tips
Remember that if you are setting up a shared queue you ought to do this separately like I have done in this example in a declare_queues.cpp. You could also use a python script.
If you need an exclusive queue for the consumer, then make sure you give it a unique name. The Qpid examples do this by adding the session ID to make it unique. So the example above actually sets the queue name used in the declare and the binding as:
queue += session.getId().getName();So my plan is to give a detailed description of these examples on the Qpid website. When that is done I'll post the links on my blog.
UPDATE
You can also browse the code for the tradedemo example here.
IP Babble is the personal blog of William Henry.
Leave a comment