IPBabble

Using AMQP with Condor for Job Submission

2009-03-10T14:52:28Z

My colleague Matt (spinningmatt) posted a really useful article, Submitting Jobs with AMQP on a Condor Based Grid. The article includes an example C++ program that uses a low-latency job submission feature of MRG.

The low-latency feature uses AMQP (MRG's Messaging) to deliver job workloads to a grid's execution nodes. This bypasses the job scheduler (Condor's schedd). Instead a special daemon on the execute node consume jobs off an AMQP queue. It's a pull versus push model.

Why is this useful? Well originally the intent was for sub-second jobs in the financial services industry. Consider a grid of index calculation applications. Each calculation may take less than a second to perform. The applications can't wait for a job scheduler to decide that their execution nodes are available, then schedule a job onto the node, then push the job out there. Instead jobs are placed on AMQP queues. As soon as an execute node is free to perform work it pulls the next job form the queue. There is barely any latency between jobs.

Of course as soon as you do this for one specific use, sub-second jobs, then others see the advantages too. This feature doesn't need to be sub-second jobs. MRG customers from various industries now see the advantage of this feature.

There are some considerations. For example, should the entire grid be untilized this way or should a specific portion of the grid be carved off for low-latency workloads? If the jobs are sub-second and high volume, should they be reported to a management console (this could cause quite a bit of clutter) or just logged? How should failed jobs be managed under different scenarios? e.g. in sub-second transactions a failed job may have missed its window of being useful and therefore there is little point in resubmitting. The answers to these quesitons will depend on the type of low-latency workload.

If you are interested in more information on this topic please remember to check out Matt's post.

AMQP and Qpid: Two Queue Examples

2009-02-16T23:08:22Z

So over the weekend I posted an entry that introduced some of the AMQP benefits. That article was very upbeat on AMQP and Qpid. Two areas of improvement are:

Potential complexity of feature configuration
Lack of concrete or industry specific examples

Improvement in the second area can really help solve the first issue.

As standards mature they can often get too complex. I remember when CORBA 2.2 came out and I read about Portable Object Adapters. I was impressed at the flexibility it provided but I was also anxious that the number of configuration options would overwhelm and discourage many developers. So while it's desirable to have lots of enhanced features, you also don't want lots and lots of features, and their combinations of configurations, to put off the novice.

Of course much of the complexity issue can be solved by providing good examples. However, examples of how to code a feature don't always give you the context. It is better if we can explain the feature with a real world example. When I developed the current C++ training examples for MRG Messaging we (at Red Hat) decided to try and build examples around a banking use case. And recently I had an example called "tradedemo" committed to the Qpid project.
]]>

Time-To-Live (TTL)
Last Value Queue (LVQ)

TTL is essentially about putting a timeout on a message. The current operation is on a specific message. i.e. each time you want this feature you must set it on the message being published. Time-To-Live is used in other technologies like IP. In IP, a packet may have a TTL associated with the number of hops on the network. But like IP, AMQP associates the TTL on the message (packet). (More on this later.)

A use case for this might be where several consumers are subscribed to the same queue (shared) and each wants to consume all messages. E.g. if several consumers were subscribed for specific stock quotes. Because none of the consumers are actually acquiring the message off of the queue, the message will remain on the queue. This is not desirable because the queue will fill up. Putting a TTL on the message means that it will be removed after a certain period of time. But how long? Well that depends. New subscribers might want to get some history of quotes for that stock. However obviously anything "too old" is of no interest. So perhaps a TTL of 3-5 seconds makes sense for this use case.

  unsigned int ttl_time = 4000;     // 4 seconds
  message.getDeliveryProperties().setTtl(ttl_time);

This brings up another issue. One can imagine that unlike IP, AMQP messages can be consumed by different types of user. Putting a TTL on the message itself might not make sense. E.g. one consumer might want to subscribe and receive all quotes for a stock for the entire day to examine trends etc. However another type of use may just be interested in the most recent quotes (i.e. within 2 seconds). Therefore it is worth considering placing TTL on a queue rather than a message. This has not being implemented yet in Qpid and I don't believe it is part of the AMQP specification.

So, to summarize, why use TTL? Well for messages that either won't be delivered for some reason or that will be shared and therefore browsed and not consumed. Both of these mean the message won't be acquired (in AMQP terms) - one by design the other by exception.

LVQ has a similar yet different use case. Last Value Queue is exactly that, it contains the last message for a particular routing key.

A use case might be market data. A consumer might not be interested in previous volume, high low prices, and market cap. figures. The history is relevant to that consumer. Rather the consumer just wants the last value. So if market data for a particular symbol (key) is updated frequently the consumer only wants the latest values. (So though TTL sounds like it could achieve this, LVQ is much better.)

  QueueOptions qo;
  qo.setOrdering(LVQ);
  std::string binding = "MRKT" + ".#";
  session.queueDeclare(arg::queue=queue, 
                       arg::exclusive=true, 
                       arg::arguments=qo);
  session.exchangeBind(arg::exchange="amq.topic", 
                       arg::queue=queue, 
                       arg::bindingKey=binding);

In this example I'm assuming an exclusive queue per consumer. Shared queues could also be used. The QueueOptions is what allows you to setup the queue as an LVQ.

You can find the tradedemo example in the C+ examples directory. Download using svn:

svn co https://svn.apache.org/repos/asf/qpid/trunk/qpid/cpp/examples/tradedemo .

There is another example of using LVQ on the Qpid site here. I am going to include a description of this demo on the Qpid site once it has been approved. I'll post when that is up there.

Some other Queue Tips
Remember that if you are setting up a shared queue you ought to do this separately like I have done in this example in a declare_queues.cpp. You could also use a python script.

If you need an exclusive queue for the consumer, then make sure you give it a unique name. The Qpid examples do this by adding the session ID to make it unique. So the example above actually sets the queue name used in the declare and the binding as:

   queue += session.getId().getName();

So my plan is to give a detailed description of these examples on the Qpid website. When that is done I'll post the links on my blog.

UPDATE

You can also browse the code for the tradedemo example here.

AMQP and Qpid

2009-02-14T20:03:18Z

This week I saw some example code I submitted to the Apache/Qpid project committed. The example demonstrates a possible use case for some Qpid queue features. I suspect we're going to see more examples with use case focus. That is a very good thing and I'd encourage Qpid developers to take time to submit such examples. We need a richer set. I'll discuss this and specifically my example on a later post.

This post is about AMQP and Qpid and is long overdue.

Look, no doubt I'm biased but the more and more I look at AMQP, the more I'm amazed with it's flexibility and it's depth and breadth of features. Its importance in the financial services industry is growing and its profile is rising in other industries (animation, energy, logistics, telecommunications etc.) and government.

What makes AMQP so important? Well there are three basic characteristics that make it compelling :

Standard wire protocol
Multi-platform, multi-language support
The model - simple, yet flexible

Standard wire protocol opens up the ecosystem and provides interoperability and freedom from vendor lock-in. We saw this with IIOP in the CORBA specification. Frankly I'm surprised and disappointed that it has taken this long to get a standard wire protocol for "messaging". I have messaging in quotes because no doubt that there are many that will point to various standards like IIOP, and HTTP and SOAP etc. Those of you that ave been working on enterprise systems with a more asynchronous messaging approach know what I mean.

Because of the standard wire-protocol it means that anyone can write on any platform or language and still take advantage of AMQP. The Qpid project currently supports C++, Java (including JMS), Python, Ruby, .NET (C#). In fact Microsoft have joined AMQP and joined Qpid! Hardware vendors like Cisco participate. The point is the ecosystem is growing.

This is very ambitious but as all these various languages and platforms with their various characteristics take advantage of AMQP? Is routing text messages through telecommunications exchanges, large image files to render farms, and low-latency high volume market data a reasonable requirement to be demanded off of one standard? That's where the flexibility comes in. Unlike other messaging systems that might focus just on a single language programming API, or on a single technology like multicast, or on a single abstraction like a queue, AMQP's model is very flexible. And yet it remains quite simple.

AMQP has three main abstractions:

An exchange - where messages are posted
A queue - where consumers find messages
A binding - that binds a specific message to one or more queues

The decoupling of message sending from message delivery makes this model very powerful. Rather than specifying a specific queue or forcing a broadcast across a network, AMQP provides a much more flexible model.

There is a nice explanation of these concepts here as part of the MRG documentation.

And so that is one of the reasons I joined Red Hat last May. AMQP and Qpid represent a new wave in enterprise application communication and I'm very excited in the interest and growth of adoption I'm seeing so far. This along with the Condor based Grid technology is a very powerful distributed computing platform for government and industry. It many ways it's what many of use were thinking would happen well over a decade ago. The growth of higher speed networks and the ubiquity of messaging in every facet of our lives are pushing technology like this to broader adoption. In the 21st century we are going to need a much more standard and ubiquitous messaging infrastructure.

Tomorrow I'll post about the example I committed to Qpid.

MRG Grid and EC2 Enhanced

2008-12-02T16:30:33Z

Last week I blogged on Using MRG Grid and EC2. Since then I've been doing some work testing what we've been calling MRG Grid with EC2 Enhanced.

Last week I was submitting an AMI as a job in the cloud. MRG Grid with EC2 Enhanced is different. In this case the user submits an ordinary (vanilla) job but we have a preference to run it in the cloud (EC2). In this case MRG Grid routes the job from the local grid to the cloud. There are some routing rules depending on the characteristics specified for cloud instances. e.g. the job may require the use of high CPU instance types or 64bit based instances.

The job actually causes AMIs (specified in the routing rules) to be launched in the cloud and then uses Web services to transfer data in and out of the running instance.

These rules are in hooks that are fired because of certain job characteristics. There are both pre and post job execution hooks. This allows for the transfer of data in and out.

Obviously the instance(s) running in the cloud need(s) an extra mechanism to get the jobs to run on it's local condor instance. There is a daemon called caroniad that listens for incoming jobs and uses the AWS Web services.

I haven't had to do anything complex yet but so far the MRG / EC2 Enhanced testing has cost be about $4!

# Example Job Submission for MRG / EC2 Enhanced
universe = vanilla
executable = /home/testuser/ec2_tests/my_ec2_job.sh
output = ec2test.out$(PROCESS)
log = ulog$(PROCESS)
requirements = Arch == "INTEL"
should_transfer_files = yes
when_to_transfer_output = on_exit
transfer_executable = true
+WantAWS = True
+WantArch = "INTEL"
+WantCPUs = 1
queue 

# More CPU requirements get a large instance type
+WantCPUs = 2
queue 2

As I mentioned in the last blog entry this will all be documented in the MRG 1.1 Grid User Guide. I've been providing some input based on my testing.

Rob Rati is the Red Hat engineer behind caroniad and he's been a great help getting me through the testing.

Using MRG Grid and EC2

2008-11-24T23:13:40Z

I've been testing some MRG Grid capabilities recently. Last week it was using MRG Grid with virtualization and also using MRG Grid with Amazon's EC2.

Using Condor's ability to work with virtualization I was able to bring up a Xen image as a job on my Grid. This image could have been a complete stack including an application. I was happy enough just to get an existing RHEL 5.2 guest running.

Toward the end of the week I was asked to look at our EC2 integration and take it for a test run. My first goal was just to get MRG Grid to launch an image in the cloud. After I created a new AWS (Amazon Web Services) account I searched and found a Fedora 6 image already up in EC2 so I decided I'd use that as my first test. That way I didn't have to go through the task of building my own AMI (Amazon Machine Image).

Next I looked at what changes I'd need to make to MRG Grid's configuration on my laptop - i.e. the local Condor configuration. The plan was that I'd launch my jobs from my laptop into the cloud. It turns out that the current MRG Grid configuration (for MRG 1.1 release in December) already has all the default configuration I need in the global configuration file. So I really had everything that I needed already, in terms of configuration.

Then I had to create a job submission file in order to have MRG Grid's scheduler (Condor's schedd) schedule the job in the EC2 cloud. I used something very similar to the folllowing:

# Note to submit an AMI as a job we need the grid universe
Universe = grid
grid_resource = amazon

# Executable in this context is just a label for the job
Executable  = my_amazon_ec2_job
transfer_executable = false

# Keys provided by AWS
amazon_public_key = cert-ABCDEFGHIJKLMNOPQRSTUVWXYZ.pem
amazon_private_key = pk-AABBCCDDEEFFGGHHIIJJKKLLMMNNOOPP.pem

# The AMI ID
amazon_ami_id = ami-123a456b
amazon_user_data = Hello EC2!

# The keypair file if needed for using ssh etc
amazon_keypair_file = /tmp/keypair
# The security group for the job
amazon_security_groups = MY_SEC_GRP

queue

The key's I got from my AWS account. The amazon_ami_id is the identifier of an AMI already loaded at EC2. And the security group is only needed if you want to open up certain ports (like 22 for ssh) on the image. All the values for keys, ids and security groups above are merely examples.

I then submitted the job and checked using the EC2 command line tools (I had downloaded earlier) and also using Elasticfox - a Firefox plug-in that provides a web condole to view and manage your AMIs.

I did a couple of other steps setting up that security group (above) with the command line tools so that I could ssh into the image. I have to say that running ssh to login to the running image in the cloud made it all seem pretty cool.

I then decided that I would write up my steps and submit them to the MRG Grid User Guide technical writer (God bless her). So I wrote up the description of the basic steps of using MRG with EC2. So when MRG 1.1 is released you'll find a more detailed decription of the steps in the MRG Grid User Guide here.

This week I have to work on the "Enhanced" MRG/EC2 integration. This is where I take advantage of a running AMI in the cloud and use it as a MRG Grid execute node and therefore deploy jobs into it. Thus making it just another resource available to my Grid, even though it is running externally in the cloud.

Oh, by the way all the EC2 testing on Thursday night and Friday morning cost be about $0.50. That will be billed to my Amazon account.

Whitehurst on Cloud, Grid, Virtualization, Open Source

2008-10-20T16:21:20Z

There is a great interview with Jim Whitehurst on ZDNet yesterday that starts on the topic of Cloud computing and virtualization and ends up speaking about the merits of open source. Jim's ability to explain the value of these innovations is really clear and to the point.

On the topic of Cloud, Jim explains that he believes that the economics of server farms will win out. He then says "whether they are internal private grids or clouds, or whether those are public, or semi-private, that's still to be determined, because that's a lot around specific business models." Therefore flexibility to use private grids or clouds or external clouds will be come very important for many companies. This flexibility is one of the Objectives of MRG Grid.

Jim then does a great job of explaining how virtualization is becoming a commodity and should be moved to the operating system layer. Then he explains the Red Hat business model for open source. Again, he makes it very concise and clear. Worth a read.

Now Available at Red Hat

2008-10-09T15:53:17Z

My blog difficulties have delayed this blog entry. I wanted to do a "Farewell IONA" blog (and I might do something later) and then an introduction to my new job at Red Hat.

After over 11 years at IONA (my couple of years at Aurora Technologies are officially included) I moved to Red Hat in May of this year. Not long after my move I started hearing rumors of IONA's acquistion. And, of course, now IONA is part of Progress Software.

My move was prompted much less by IONA's fate than by what Red Hat is doing with MRG. It was time for change and MRG represents something very exciting going on in the industry. There is a lot I can and will be saying about MRG on this IPBabble and I'll only give a short introduction in this entry.

Red Hat's expansion into emerging technologies is an effort to provide more value to its customers. This work includes, among other things, virtualization, management and Messaging, Realtime and Grid (MRG). I'll be explaining more about details in later posts but briefly:

Messaging - an open source (Qpid), open standard (AMQP) based, platform independent, messaging infrastructure.
Realtime - a predictable, low-latency linux kernel.
Grid - an open source (Condor) HPC, HTC platform that provides the flexibility of local grids, remotes grids and cloud computing.

Of course there are lots more details but that is the very high level view. I'll be diving in a lot more on ipbabble.com.

I have a hybrid role: I work for Brian Stevens, the CTO, in the Office of CTO focused on MRG. In that way I'm very much an evangelist and customer focused solution architect. But in the same organization I'm an engineer working on MRG. So I have to contribute to the engineering effort too. It is both a very rewarding and very challenging role. Getting up to speed on all the various aspects of MRG has been a steep learning curve despite my background in distributed computing and messaging. Thankfully I work with a really smart and dedicated team that has been generous with sharing their knowledge. I'm sure I'll be introducing some of them on this blog in the future.

UPDATE: Thank you to Lana for spotting the "bog" typo.

Moving blog issues

2008-10-09T00:10:07Z

Well I have this new blog set up but I can't figure out how to move all my old blog entries into this blog. I guess I didn't export them correctly and just zipped up the files. If anyone knows how to get old Movable Type based blog entries into a new version of MT then let me know.

Update:
I'm just loading these manually and setting the date to the original published date. I doubt I'll get around to entering the comments and trackbacks for the old entries. I'll have to see.

IP Babble Will BE Available Again Soon

2008-09-29T02:32:32Z

I have moved IPBabble.com's hosting service and I'm trying to move all my previous content over.

Thank you for your patience.

More RESTful clarity

2007-07-18T04:43:28Z

So I talked, well actually chatted via instant-messaging, with Steve Vinoski and I do need to clear up some points from my last blog entry.

Steve pointed out that RPC does NOT require synchronous communications. This is true. One can simulate the request-response of a remote procedure using asynchronous communications. But the idea is that RPC appear synchronous - like a programming language procedure that blocks.

Some synchronous technology can appear asynchronous. E.g. CORBA has oneway operations. And many queue based (messaging) technologies use connection based technology underneath. E.g. Some publish subscribe technologies are actually connection based. The publisher connects to the notification service and so does the subscriber when retrieving messages. They are asynchronous to the application in the sense that they are decoupled.

]]> Back to the Steve conversation. Steve maintained that RPC basically implied location transparency. I'm not sure I agree. Though many RPC technologies have location transparency I don't think that that the orginal intent for RPC was that it have location transparency. I certainly regard come technologies today as RPC that don't have location transparency. RFC 707 doesn't mention location transparency and as it works through command/response, request/reply and procedure call models it speaks of making it easier but I don't see transparent. In fact in order not to "mislead the application programmer" the paper suggests the model be clarified with "Local procedure calls are cheap; remote procedure calls are not."

Location transparency is very often a bad thing. (more often than not as we discovered with novice developers of CORBA and distributed Java technologies and many more.) Because location transparency hides the network, this can mislead the developer and also introduces issues of extra exception handling that the developer has make sure that they cover. Steve wrote an excellent paper on some of the issues in IEEE's Internet Computing back in late 2005 - I have a copy but I don't have a public URL for it, sorry.

But one could argue this both ways. i.e. The technology is not good because it misleads the programmer OR you could say that one needs better than average programmers to develop distributed application architectures using RPC. But it seems that one could say the same thing about RESTful developers. So before we accept the former (the technology is not good) let us remember that not all REST developers are building truly RESTful based applications/services. In fact RESTful Web services points out that most REST based services are REST-RPC and not RESTful.

Again, I wish to point out that I continue to warm to REST and Resource Oriented Architectures, I see the potential, and look forward to implementing. I'm just not willing to say that REST is useful in all situations or that SOAP and WS-* are evil and there is no place for them. And, as the book points out, there is still a lot of confusion with REST based on the wide deployment of REST-RPC services rather than RESTful.

The book has this to say about breaking the uniform interface: "occasionally it's unavoidable" (p101). Even while it's undesirable. The section on POST describes this scenario. And so some "overloading" is required. Does this mean then that REST is broken? Or should the REST community loosen the definition of RESTful? Is "reshuffling" the resources acceptable, or a hack that demonstrates something that needs to be addressed with the model?

RESTful Myths: Unraveling the Confusion

2007-07-16T16:55:02Z

Shame on me for not delving into this deeper, sooner. It was high time I took on some of the myths in the REST community again.

First let me say that I like REST. I think the architecture makes a lot of sense in many cases and I look forward to implementing my first RESTful services(s). Having said that the REST community don't do themselves many favors with the enterprise community, especially those that have been around the block several times and have built several flavors of architectures over the years. We know our architectures -we know our terminology. It seems the REST guys confuse some terms. So I'm going to target some areas to help unravel the muddle.

]]> I've decided to base this critique on the new popular book RESTful Web services by Richardson & Ruby. I'm assuming that the REST community stand by this book - otherwise, sadly, it just adds some more confusion. To the authors I say thank you, I'm enjoying the book. Don't think I don't like it just because of this blog. It has not only helped me appreciate some of the confusion in the industry but it has helped me to appreciate the RESTful resource oriented approach.

Before I start I'll summarize by saying this. Perhaps much of the present confusion is that the REST community has changed the semantics of some popular enterprise computing terminology. We shall address these below.

Myth # 1: (from page 19) "All Web services are message-oriented, because HTTP itself is message oriented." One could argue, in a very loose way, that HTTP is message oriented but it's a bit like saying that airplanes are "ground transport" just because they can travel on the ground. Modern convention says that airplanes are air transport. Modern, popular conventions says that message oriented is asynchronous not synchronous. HTTP is a synchronous transport. Message Oriented Middleware (MOM) is a class of technology that includes JMS, MQ Series, CORBA's Event Service, etc. Generally regarded as connectionless technology. So though one could regard HTTP as message oriented in some general english language sense, those in the industry a long time would not call HTTP "message oriented". HTTP is a connection based, synchronous technology. Nothing wrong with that but it is interesting considering some of the later myths which criticize RPC style technology, where WS-* is singled out.

Myth # 2: REST is not RPC and RPC is in some way bad. When reading this book and watching or participating in discussions on REST there is implicit and explicit criticism of RPC based approaches. But again there is confusion here. The REST folk mean that having an explicit operation/method name in a message or in the invocation is what is meant by RPC. However there are two or three ways that this argument doesn't work and is confusing.

Again a terminology confusion. Most people, that have been in the industry for some time, usually refer to RPC when talking about synchronous, connection based communication. The emphasis is on that rather than a rich vocabulary of method names - though that is part of the assumption.
HTTP is an RPC model. It's synchronous and connection based. Moreover, it has a relatively rich set of remote procedure vocabulary - POST, PUT, GET, DELETE being some of them. Question: So if someone using CORBA used DII and had an interface with only three or four generic operations like HTTP, would that make it RESTful?

Myth # 3: REST makes things simpler by having a uniform generic interface. There are two ways to approach this. Again I'd put the example of MOM based communications (e.g. JMS), or DII in CORBA. All of these technologies have a uniform interfaces. CORBA has invoke, send_oneway, send_deferred,etc. JMS has publish. Does a uniform interface then make it simpler? Not necessarily because then the application code needs to have more smarts in it for marshaling and message validation etc. The static interfaces defined by using CORBA IDL based proxies or Java interfaces with RMI helped remove this extra work "Let the middleware worry about that," was a reason to use such things. But this is not pure enough for REST. But for most Web facing applicaitons REST works great! But wait a minute, what's this? From page 25: "Instead, as a smart programmer you'll quickly notice the patterns underlying your requests to a given service, and write wrapper methods that abstract away the details of HTTP access." Surely not! Surely this is blasphemy! Could the RESTful be advocating some sort of wrapper proxies? Sounds like it. Hmmmm .... I've said this before in a blog entry called I've been arRESTed! HAving said that I had far less a clue about REST then than I do now.

So some questions for the REST folks:

If I was using JMS and had a different queue defined for each resource and had four messages defined: add, delete, update, and get, and I sent XML payloads using these queues, is that RESTful? By definition it would seem that it is.

I have to conclude the following, just as the REST folks say that Web services and SOAP is hijacked by WS-*, is REST hijacked by HTTP? (In fairness the book does not give out to much about SOAP just the way that most implementations use SOAP.) I mean can RESTful services be implemented using another transport? And can a uniform interface have operations similar but different to HTTP's POST, PUT, DELETE and GET? Is it okay if they map to CRUD?

What about a service where the semantics of HTTP verbs aren't enough? Is REST only useful in it's purest RESTful way for resources that basically only have CRUD interface? What about objects that have richer semantics? Does the semantics (and therefore the verb) get hidden in the data of a PUT? (And therefore break the rules) Or do we have to define more finer resources in order to fit the REST model?

Now at the start I made it clear that I like REST. I do! In fact I think it is really neat. I think it has lots of applications. Again I look forward to implementing a resource oriented architecture using REST. And I'll do it just like Richardson and Ruby suggest. It makes sense.

The bottom line is that REST is a really cool HTTP based and CRUD based architecural style that works great for web based applications. It makes one more interface assumption that helps it a great deal - It assumes that most data is text and so lots of the ugly marshalling issues that would push you towards using other technologies go out the window. You don't need them and therefore REST makes sense. So it makes communiction simple and then you can rely on high speed processors to do any marshalling into various binary types that you might be using in your application code. Again nothing wrong with this for a great deal of applications.

My challenge is that the REST community work with the other communities, including WS-* (the REST community has cute nicknames for this community) to help clear up some of the confusion - and I think some are trying. For example I don't think it's fair to assume that Web services is just HTTP. HTTP is successful for a number of reasons and not just the fact that it has a simple, uniform interface. Remember most enterprise architects got stuck with HTTP because web administrators didn't like opening up lots of ports - no matter how secure the alternative transport/protocol might be. As a result people were doing ugly things like tunneling through HTTP to do B2B. As a result we just had to resign ourselves to the fact that HTTP was the "Web" when it came to outside the firewall. But there is no reason that alternative approaches cannot be used, especially behind the firewall.

And so I think it is important to define some boundaries to where REST works really well and where it doesn't. Let's be honest about it an not have so many ideological wars. REST is really really useful ... so are other approaches.

My expectation is that the response I'll get to this is "he just doesn't get it!" coupled with the "REST can do everything" answer instead of some clarification, some constructive suggestion on how the REST community can be more inclusive while at the same time remaining RESTful, and some best practices. This would not only do a good service to the larger community, as the RESTful Web services book has done a good service educating me, but it would also help the REST community itself because, as the book points out, most REST implementations are not RESTful but "REST-RPC" based. So there is plenty of confusion currently and room for improvement.

I'm not claiming to be a REST expert. I'm not claiming REST is bad. I just want some honest debate and less confusion around the terms. There is a danger that REST might "go to rest" because there is confusion and disagreement. I hope to learn more about REST as I continue to read the book (no doubt some of the above questions are answered in sections I haven't reached yet.)

Vista & After The Love Is Gone

2007-07-11T17:04:00Z

This article should have started "sitting here at the Ballmer's keynote at Microsoft's Worldwide Partner Conference in Denver". Instead, I'm writing it from my home office a day late before I head back up.

Before I start I need to point out that I work with Microsoft in their partner program. I'm committed to that. Microsoft technologies and products play a big part in the enterprise and will continue to do so. It's not going away. Microsoft often have some "negative" things to say about certain products etc. of their partners so I think it's okay for me to be critical if it's constructive and can lead to better products. (Also see the disclaimer about these being my opinions and not that of my company on the right ;-)

]]> Yesterday I was listening to Mr. Ballmer talk about "optimized desktops" and "enabling people to realize their potential" (never mind that the promise of technology to help people realize their potential seems a bit of a stretch). While I was listening I was shaking my head in disappointment. I had wanted this to be a positive upbeat, experience but what had just happened to me had already soured my perspective.

You see I decided to leave my Mac at home and be a good corporate citizen and take my newer Dell/Microsoft Vista laptop. Sigh, what a mistake. Those of you that own a Mac, know how easy it is to just connect to the internet, especially on WiFi. Well I had being in the main conf. dining area with my Vista laptop trying to connect for an hour. I've seen colleagues with this problem before. Now I was experiencing it first hand. You see I made the mistake of just putting my laptop to sleep and then opening it, reawakening it (always hit-or-miss IMO) and trying to connect. Well Vista couldn't see the WiFi at the conf. center. I scoured the conference information book to see if there was some SSID I was supposed to be using. I then decided that perhaps the Colorado Conference Center was a little behind and I had to be somewhere special Microsoft's Blue Lounge for WiFi access. I headed outside the dining area.

As I left the dining area I noticed some people sitting around with laptops and asked if they were online. "Sure" they said. hmmmm... Of course! I remembered. This is still Windows! See my problem was that I expected more from Vista in this area. But I needed to reboot. Of course, I couldn't wake my laptop from sleep - hey, sometimes it wakes sometimes it doesn't you just don't know. I tried everything and then I decided that I was going to reboot anyway I might a well do the horrible hard boot by holding down the power button - not elegant but some things don't change. (For those of you shutting "user error, user error" remember I have two degrees in computer science (B.Sc. and M.Sc.) and have been working on computers since before 1984 in some way or other. I've programmed Windows, Solaris, Cray, IBM Mainframe, HP/UX, AIX, OS X, Linux .....)

After the hard reboot sure enough there was the WiFi hotspot MSFTINET. Yea! I connected! But woops, gotta go into see Mr. Ballmer's keynote. Hopefully the laptop will remember MSFTINET. Well when I got around to the overflow room - my delay contributed to missing the main hall - I discovered that MSFTINET was not visible from here. But that's fine there was MSFTEVENT and MSFTWAN and some others. Forget it. No luck. I closed my laptop disappointed and instead watched Mr. Ballmer jump around the stage like a rock star. Fair play to him - he's a lot to be excited about.

Ballmer talked about how Microsoft was once a desktop company and still is but that was once perceived as not-an-Enterprise-company. However now they are told how they can be a "better enterprise company", inferring that they are at least an enterprise company now. I think that is very fair- there is no denying that fact. Microsoft are a very strong enterprise company. Sun and HP have slipped up in this area. IBM are still very strong. But I couldn't help thinking, "you could still be a better desktop/laptop company". Now maybe I'm being unfair perhaps this is a Dell problem.

So enough about the negative experience. Here are some of the positives. Ballmer introduced Sliverlight a "cross-browser, cross-platform plug-in for delivering the next generation of Microsoft .NET-based media experiences and rich interactive applications for the Web." The technology walkthrough and demo, performed by someone else (I'm ashamed I can't remember their name) was very impressive. Actually I decided to download it on my Mac today to see it work. Some bad news for Microsoft - I promise this was supposed to be a positive section - it wouldn't install on Safari. The install reported that I required "10.4.8 or higher" to install and so it wouldn't install. Well I have 10.4.10!!! Come on guys (MS)!!! I wanted to try this out and say really positive stuff! Anyway I'm sure this is a small glitch. I'll try it on Vista later or on Firefox. It looked really slick.

Microsoft also talked about the Cloud Platform - a place for developing new service based applications etc. Hmmm... we (at IONA) have been talking about the cloud for about 3 years. Sounds very familiar.

So I really need to say something completely positive with no negative strings. Ah yes, I saw Surface. I had seen a YouTube demo of this technology (or like technology ? ) some years ago. It is so very very cool. Honestly it's the type of technology and product you'd expect from Apple. Congratulations to Microsoft for swooping this up. Reader, if you haven't seen this yet check out the cool promotion demos here. (Skip the introduction).

So tonight, if they'll still let me, I'll be seeing Earth Wind and Fire at one of the Microsoft events. Oh you thought that "After the Love is Gone" was about Vista. No, it's one of my favorite EW&F hits.

Update: Sigh. So it's not Earth Wind and Fire but instead Earth Wind Fire and Water, a cover/tribute band. Did someone say, After the Love is Gone?

Update: Went to a Silverlight presentation and afterwards mentioned the trouble with downloading Silverlight on Safari (I said it privately not in the general Q&A) The guy blew me off with "well it's beta!". Well don't waste my time! It could easily have checked my browser and said not to bother downloading cause it doesn't really support OS X in beta. (BTW it wouldn't load for Firefox on OS X either) Instead I went through the download process to get the crazy message. BTW the Silverlight demo called Top Banana was really cool. Beau from Metaliq said that the development was remarkably easy. They did say that Silverlight will not support WCF.

Last Week SOCA. This Week Catalyst

2007-06-27T23:27:53Z

Last week I was in L.A. talking about SOA on an expert panel at SOCA 2007. It was an honor to be on the panel. I really enjoyed Prof. Wei-Tek Tsai's discussion about how the process for developing SOA has not yet being completely figured out yet. He claims there was a big shift in methodologies and processes when we went to OO and that a similar shift is needed in SOA. I mentioned some of my experience in this shift in a previous blog called Trusted Services Network. I hope to catch up with Prof. Tsai regarding this in the future.

Today I've just finished my presentation on OSGi at Burton Group's Catalyst conference in San Francisco. My presentation was well received. I'll blog on OSGi soon. I will say that it was amazing how complimentary the presentations in my session were. It was almost as if we had rehearsed the message but we had not.

One of the cool presentations was Jeff Barr's from Amazon regarding their web services. His discussion on Amazon's approach to web-Scale Computing really was an eye opener. I'm embarrassed I didn't know about this before. I knew that Amazon was providing Web service APIs for developers but I had no idea that they had a complete infrastructure for organizations to utilize. Hardware and infrastructure as a service. This is not unlike what some of the large telecommunications companies are trying to do - i.e. provide their infrastructure for people to deploy their services. Amazon is providing that and more - the ability to host your entire IT on their "network".

Scratch Programming Language

2007-06-06T19:33:50Z

I was reading a post on Feld Thoughts about an new programming language developed at MIT called Scratch. It sounded good, so I thought I better download it and have a look at it to see if it was suitable for my eight year old daughter - she's very bright and I've been trying to find a way to introduce her to programming for a while but couldn't find the right tool.

I downloaded Scratch and she was up and playing with it in no time at all. She created an animation with her own sprites and had them moving around the screen.

The developers of Scratch clearly put a lot of thought into it. What's cool about Scratch:

It has it's own very cool IDE. This includes color coded building blocks for different language construct categories.
The building blocks make it very easy for young kids but they aren't too childish so that even an adult can enjoy playing with it.
By building in the easy to use animation tools, they have made it fun from the start.
There is a community growing around it from their site where kids can share their creations and use and comment on other people's creations.

My daughter enjoyed it so much she now has the other kids in the neighborhood downloading and playing with it. And I have other friends and colleagues who have downloaded Scratch for their kids.

The website is full of some really neat programs created using Scratch ,including this Archery game. (Its only drawback being the annoying Scarborough Fair background music).

The ServerSide Video

2007-06-05T03:42:03Z

I mentioned in a previous post called TheServerSide Symposium Las Vegas that I was on a SOA expert panel at the show. Well I recieved the link to the video the other day and thought I'd better post it. Click on "Launch Presentation" on SOA Panel from TheServerSide Symposium. The transcript is also available on the launched panel through a tab.

I won't give you my own opiniion of the panel. You can judge for yourself. Feedback welcome and appreciated. If you have trouble viewing it please let me know too.