# Example Job Submission for MRG / EC2 Enhanced
universe = vanilla executable = /home/testuser/ec2_tests/my_ec2_job.sh output = ec2test.out$(PROCESS) log = ulog$(PROCESS) requirements = Arch == "INTEL" should_transfer_files = yes when_to_transfer_output = on_exit transfer_executable = true +WantAWS = True +WantArch = "INTEL" +WantCPUs = 1 queue
# More CPU requirements get a large instance type +WantCPUs = 2 queue 2
The key's I got from my AWS account. The amazon_ami_id is the identifier of an AMI already loaded at EC2. And the security group is only needed if you want to open up certain ports (like 22 for ssh) on the image. All the values for keys, ids and security groups above are merely examples.# Note to submit an AMI as a job we need the grid universe
Universe = grid
grid_resource = amazon
# Executable in this context is just a label for the job
Executable = my_amazon_ec2_job
transfer_executable = false
# Keys provided by AWS
amazon_public_key = cert-ABCDEFGHIJKLMNOPQRSTUVWXYZ.pem
amazon_private_key = pk-AABBCCDDEEFFGGHHIIJJKKLLMMNNOOPP.pem
# The AMI ID
amazon_ami_id = ami-123a456b
amazon_user_data = Hello EC2!
# The keypair file if needed for using ssh etc
amazon_keypair_file = /tmp/keypair
# The security group for the job
amazon_security_groups = MY_SEC_GRP
queue
So I talked, well actually chatted via instant-messaging, with Steve Vinoski and I do need to clear up some points from my last blog entry.
Steve pointed out that RPC does NOT require synchronous communications. This is true. One can simulate the request-response of a remote procedure using asynchronous communications. But the idea is that RPC appear synchronous - like a programming language procedure that blocks.
Some synchronous technology can appear asynchronous. E.g. CORBA has oneway operations. And many queue based (messaging) technologies use connection based technology underneath. E.g. Some publish subscribe technologies are actually connection based. The publisher connects to the notification service and so does the subscriber when retrieving messages. They are asynchronous to the application in the sense that they are decoupled.
Location transparency is very often a bad thing. (more often than not as we discovered with novice developers of CORBA and distributed Java technologies and many more.) Because location transparency hides the network, this can mislead the developer and also introduces issues of extra exception handling that the developer has make sure that they cover. Steve wrote an excellent paper on some of the issues in IEEE's Internet Computing back in late 2005 - I have a copy but I don't have a public URL for it, sorry.
But one could argue this both ways. i.e. The technology is not good because it misleads the programmer OR you could say that one needs better than average programmers to develop distributed application architectures using RPC. But it seems that one could say the same thing about RESTful developers. So before we accept the former (the technology is not good) let us remember that not all REST developers are building truly RESTful based applications/services. In fact RESTful Web services points out that most REST based services are REST-RPC and not RESTful.
Again, I wish to point out that I continue to warm to REST and Resource Oriented Architectures, I see the potential, and look forward to implementing. I'm just not willing to say that REST is useful in all situations or that SOAP and WS-* are evil and there is no place for them. And, as the book points out, there is still a lot of confusion with REST based on the wide deployment of REST-RPC services rather than RESTful.
The book has this to say about breaking the uniform interface: "occasionally it's unavoidable" (p101). Even while it's undesirable. The section on POST describes this scenario. And so some "overloading" is required. Does this mean then that REST is broken? Or should the REST community loosen the definition of RESTful? Is "reshuffling" the resources acceptable, or a hack that demonstrates something that needs to be addressed with the model?
]]>Shame on me for not delving into this deeper, sooner. It was high time I took on some of the myths in the REST community again.
First let me say that I like REST. I think the architecture makes a lot of sense in many cases and I look forward to implementing my first RESTful services(s). Having said that the REST community don't do themselves many favors with the enterprise community, especially those that have been around the block several times and have built several flavors of architectures over the years. We know our architectures -we know our terminology. It seems the REST guys confuse some terms. So I'm going to target some areas to help unravel the muddle.
Before I start I'll summarize by saying this. Perhaps much of the present confusion is that the REST community has changed the semantics of some popular enterprise computing terminology. We shall address these below.
Myth # 1: (from page 19) "All Web services are message-oriented, because HTTP itself is message oriented." One could argue, in a very loose way, that HTTP is message oriented but it's a bit like saying that airplanes are "ground transport" just because they can travel on the ground. Modern convention says that airplanes are air transport. Modern, popular conventions says that message oriented is asynchronous not synchronous. HTTP is a synchronous transport. Message Oriented Middleware (MOM) is a class of technology that includes JMS, MQ Series, CORBA's Event Service, etc. Generally regarded as connectionless technology. So though one could regard HTTP as message oriented in some general english language sense, those in the industry a long time would not call HTTP "message oriented". HTTP is a connection based, synchronous technology. Nothing wrong with that but it is interesting considering some of the later myths which criticize RPC style technology, where WS-* is singled out.
Myth # 2: REST is not RPC and RPC is in some way bad. When reading this book and watching or participating in discussions on REST there is implicit and explicit criticism of RPC based approaches. But again there is confusion here. The REST folk mean that having an explicit operation/method name in a message or in the invocation is what is meant by RPC. However there are two or three ways that this argument doesn't work and is confusing.
Myth # 3: REST makes things simpler by having a uniform generic interface. There are two ways to approach this. Again I'd put the example of MOM based communications (e.g. JMS), or DII in CORBA. All of these technologies have a uniform interfaces. CORBA has invoke, send_oneway, send_deferred,etc. JMS has publish. Does a uniform interface then make it simpler? Not necessarily because then the application code needs to have more smarts in it for marshaling and message validation etc. The static interfaces defined by using CORBA IDL based proxies or Java interfaces with RMI helped remove this extra work "Let the middleware worry about that," was a reason to use such things. But this is not pure enough for REST. But for most Web facing applicaitons REST works great! But wait a minute, what's this? From page 25: "Instead, as a smart programmer you'll quickly notice the patterns underlying your requests to a given service, and write wrapper methods that abstract away the details of HTTP access." Surely not! Surely this is blasphemy! Could the RESTful be advocating some sort of wrapper proxies? Sounds like it. Hmmmm .... I've said this before in a blog entry called I've been arRESTed! HAving said that I had far less a clue about REST then than I do now.
So some questions for the REST folks:
If I was using JMS and had a different queue defined for each resource and had four messages defined: add, delete, update, and get, and I sent XML payloads using these queues, is that RESTful? By definition it would seem that it is.
I have to conclude the following, just as the REST folks say that Web services and SOAP is hijacked by WS-*, is REST hijacked by HTTP? (In fairness the book does not give out to much about SOAP just the way that most implementations use SOAP.) I mean can RESTful services be implemented using another transport? And can a uniform interface have operations similar but different to HTTP's POST, PUT, DELETE and GET? Is it okay if they map to CRUD?
What about a service where the semantics of HTTP verbs aren't enough? Is REST only useful in it's purest RESTful way for resources that basically only have CRUD interface? What about objects that have richer semantics? Does the semantics (and therefore the verb) get hidden in the data of a PUT? (And therefore break the rules) Or do we have to define more finer resources in order to fit the REST model?
Now at the start I made it clear that I like REST. I do! In fact I think it is really neat. I think it has lots of applications. Again I look forward to implementing a resource oriented architecture using REST. And I'll do it just like Richardson and Ruby suggest. It makes sense.
The bottom line is that REST is a really cool HTTP based and CRUD based architecural style that works great for web based applications. It makes one more interface assumption that helps it a great deal - It assumes that most data is text and so lots of the ugly marshalling issues that would push you towards using other technologies go out the window. You don't need them and therefore REST makes sense. So it makes communiction simple and then you can rely on high speed processors to do any marshalling into various binary types that you might be using in your application code. Again nothing wrong with this for a great deal of applications.
My challenge is that the REST community work with the other communities, including WS-* (the REST community has cute nicknames for this community) to help clear up some of the confusion - and I think some are trying. For example I don't think it's fair to assume that Web services is just HTTP. HTTP is successful for a number of reasons and not just the fact that it has a simple, uniform interface. Remember most enterprise architects got stuck with HTTP because web administrators didn't like opening up lots of ports - no matter how secure the alternative transport/protocol might be. As a result people were doing ugly things like tunneling through HTTP to do B2B. As a result we just had to resign ourselves to the fact that HTTP was the "Web" when it came to outside the firewall. But there is no reason that alternative approaches cannot be used, especially behind the firewall.
And so I think it is important to define some boundaries to where REST works really well and where it doesn't. Let's be honest about it an not have so many ideological wars. REST is really really useful ... so are other approaches.
My expectation is that the response I'll get to this is "he just doesn't get it!" coupled with the "REST can do everything" answer instead of some clarification, some constructive suggestion on how the REST community can be more inclusive while at the same time remaining RESTful, and some best practices. This would not only do a good service to the larger community, as the RESTful Web services book has done a good service educating me, but it would also help the REST community itself because, as the book points out, most REST implementations are not RESTful but "REST-RPC" based. So there is plenty of confusion currently and room for improvement.
I'm not claiming to be a REST expert. I'm not claiming REST is bad. I just want some honest debate and less confusion around the terms. There is a danger that REST might "go to rest" because there is confusion and disagreement. I hope to learn more about REST as I continue to read the book (no doubt some of the above questions are answered in sections I haven't reached yet.)
Before I start I need to point out that I work with Microsoft in their partner program. I'm committed to that. Microsoft technologies and products play a big part in the enterprise and will continue to do so. It's not going away. Microsoft often have some "negative" things to say about certain products etc. of their partners so I think it's okay for me to be critical if it's constructive and can lead to better products. (Also see the disclaimer about these being my opinions and not that of my company on the right ;-)
You see I decided to leave my Mac at home and be a good corporate citizen and take my newer Dell/Microsoft Vista laptop. Sigh, what a mistake. Those of you that own a Mac, know how easy it is to just connect to the internet, especially on WiFi. Well I had being in the main conf. dining area with my Vista laptop trying to connect for an hour. I've seen colleagues with this problem before. Now I was experiencing it first hand. You see I made the mistake of just putting my laptop to sleep and then opening it, reawakening it (always hit-or-miss IMO) and trying to connect. Well Vista couldn't see the WiFi at the conf. center. I scoured the conference information book to see if there was some SSID I was supposed to be using. I then decided that perhaps the Colorado Conference Center was a little behind and I had to be somewhere special Microsoft's Blue Lounge for WiFi access. I headed outside the dining area.
As I left the dining area I noticed some people sitting around with laptops and asked if they were online. "Sure" they said. hmmmm... Of course! I remembered. This is still Windows! See my problem was that I expected more from Vista in this area. But I needed to reboot. Of course, I couldn't wake my laptop from sleep - hey, sometimes it wakes sometimes it doesn't you just don't know. I tried everything and then I decided that I was going to reboot anyway I might a well do the horrible hard boot by holding down the power button - not elegant but some things don't change. (For those of you shutting "user error, user error" remember I have two degrees in computer science (B.Sc. and M.Sc.) and have been working on computers since before 1984 in some way or other. I've programmed Windows, Solaris, Cray, IBM Mainframe, HP/UX, AIX, OS X, Linux .....)
After the hard reboot sure enough there was the WiFi hotspot MSFTINET. Yea! I connected! But woops, gotta go into see Mr. Ballmer's keynote. Hopefully the laptop will remember MSFTINET. Well when I got around to the overflow room - my delay contributed to missing the main hall - I discovered that MSFTINET was not visible from here. But that's fine there was MSFTEVENT and MSFTWAN and some others. Forget it. No luck. I closed my laptop disappointed and instead watched Mr. Ballmer jump around the stage like a rock star. Fair play to him - he's a lot to be excited about.
Ballmer talked about how Microsoft was once a desktop company and still is but that was once perceived as not-an-Enterprise-company. However now they are told how they can be a "better enterprise company", inferring that they are at least an enterprise company now. I think that is very fair- there is no denying that fact. Microsoft are a very strong enterprise company. Sun and HP have slipped up in this area. IBM are still very strong. But I couldn't help thinking, "you could still be a better desktop/laptop company". Now maybe I'm being unfair perhaps this is a Dell problem.
So enough about the negative experience. Here are some of the positives. Ballmer introduced Sliverlight a "cross-browser, cross-platform plug-in for delivering the next generation of Microsoft .NET-based media experiences and rich interactive applications for the Web." The technology walkthrough and demo, performed by someone else (I'm ashamed I can't remember their name) was very impressive. Actually I decided to download it on my Mac today to see it work. Some bad news for Microsoft - I promise this was supposed to be a positive section - it wouldn't install on Safari. The install reported that I required "10.4.8 or higher" to install and so it wouldn't install. Well I have 10.4.10!!! Come on guys (MS)!!! I wanted to try this out and say really positive stuff! Anyway I'm sure this is a small glitch. I'll try it on Vista later or on Firefox. It looked really slick.
Microsoft also talked about the Cloud Platform - a place for developing new service based applications etc. Hmmm... we (at IONA) have been talking about the cloud for about 3 years. Sounds very familiar.
So I really need to say something completely positive with no negative strings. Ah yes, I saw Surface. I had seen a YouTube demo of this technology (or like technology ? ) some years ago. It is so very very cool. Honestly it's the type of technology and product you'd expect from Apple. Congratulations to Microsoft for swooping this up. Reader, if you haven't seen this yet check out the cool promotion demos here. (Skip the introduction).
So tonight, if they'll still let me, I'll be seeing Earth Wind and Fire at one of the Microsoft events. Oh you thought that "After the Love is Gone" was about Vista. No, it's one of my favorite EW&F hits.
Update: Sigh. So it's not Earth Wind and Fire but instead Earth Wind Fire and Water, a cover/tribute band. Did someone say, After the Love is Gone?
Update: Went to a Silverlight presentation and afterwards mentioned the trouble with downloading Silverlight on Safari (I said it privately not in the general Q&A) The guy blew me off with "well it's beta!". Well don't waste my time! It could easily have checked my browser and said not to bother downloading cause it doesn't really support OS X in beta. (BTW it wouldn't load for Firefox on OS X either) Instead I went through the download process to get the crazy message. BTW the Silverlight demo called Top Banana was really cool. Beau from Metaliq said that the development was remarkably easy. They did say that Silverlight will not support WCF.
]]>Last week I was in L.A. talking about SOA on an expert panel at SOCA 2007. It was an honor to be on the panel. I really enjoyed Prof. Wei-Tek Tsai's discussion about how the process for developing SOA has not yet being completely figured out yet. He claims there was a big shift in methodologies and processes when we went to OO and that a similar shift is needed in SOA. I mentioned some of my experience in this shift in a previous blog called Trusted Services Network. I hope to catch up with Prof. Tsai regarding this in the future.
Today I've just finished my presentation on OSGi at Burton Group's Catalyst conference in San Francisco. My presentation was well received. I'll blog on OSGi soon. I will say that it was amazing how complimentary the presentations in my session were. It was almost as if we had rehearsed the message but we had not.
One of the cool presentations was Jeff Barr's from Amazon regarding their web services. His discussion on Amazon's approach to web-Scale Computing really was an eye opener. I'm embarrassed I didn't know about this before. I knew that Amazon was providing Web service APIs for developers but I had no idea that they had a complete infrastructure for organizations to utilize. Hardware and infrastructure as a service. This is not unlike what some of the large telecommunications companies are trying to do - i.e. provide their infrastructure for people to deploy their services. Amazon is providing that and more - the ability to host your entire IT on their "network".
More on Catalyst later.
BTW I do love San Francisco. What a beautiful town. Not perfect but really beautiful.
]]>I was reading a post on Feld Thoughts about an new programming language developed at MIT called Scratch. It sounded good, so I thought I better download it and have a look at it to see if it was suitable for my eight year old daughter - she's very bright and I've been trying to find a way to introduce her to programming for a while but couldn't find the right tool.
I downloaded Scratch and she was up and playing with it in no time at all. She created an animation with her own sprites and had them moving around the screen.
The developers of Scratch clearly put a lot of thought into it. What's cool about Scratch:
The website is full of some really neat programs created using Scratch ,including this Archery game. (Its only drawback being the annoying Scarborough Fair background music).
I mentioned in a previous post called TheServerSide Symposium Las Vegas that I was on a SOA expert panel at the show. Well I recieved the link to the video the other day and thought I'd better post it. Click on "Launch Presentation" on SOA Panel from TheServerSide Symposium. The transcript is also available on the launched panel through a tab.
I won't give you my own opiniion of the panel. You can judge for yourself. Feedback welcome and appreciated. If you have trouble viewing it please let me know too.
Is this article for real? I'm not sure that if I were Sun I'd have a link to an article titled "How Sun sells its SOA dog food to its own employees" on my SOA web page. But that's where I found it. Ouch! I personally don't eat dog food.
Anyway, I guess the bigger question is whether anyone else is eating this dog foo... I mean JCAPS besides Sun. And from the article it sounds like Sun had a hard time stomaching it.
This article does a good job in explaining some of the difficulties that a novice can have trying to sell SOA. Ricigliano, a senior manager at Sun tries to sell SOA internally. The article says, "he quickly found that it didn't work to go into a meeting with business users and talk about SOAP, Web services, UDDI and WSDL, or even the virtues of Sun's very own SOA tools." He learned his lesson and concluded, "the trick is not to show business users how SOA works, but how SOA can work for them." Great quote!
However the article really doesn't give evidence that Ricigliano actually implemented anything. I can only conclude from the article that they're still eating.
]]>I've been visiting several telecommunications companies over the last couple of years about the difficulty in getting to the convergence nirvana. Recently I've been discussing the development of the service network with colleagues of mine, including Tony Parker and Brian Whittaker. One of the exciting developments is the idea of opening up their networks for third party applications offered as services. It's the idea of providing your network as a marketplace for Software as a Service (SaaS).
One of the overriding themes Tony, Brian and I have been hearing lately is the idea that while the technology is more or less available for the rapid deployment of services on the network, consumers of those services will have difficulty trusting these services and the network. And service providers are wary of the impact that consumers or other service providers can have on their services. In order for the network provider to be successful the consumers of services must be confident in the quality of the services on the network. These consumers include the produces of new services based on composites (mashups) of other services.
Well formed service contracts are required because this is going to be one of the main selling points for a service. What the service provides and how it provides it, is crucial to building confidence with consumers. The design of new services or the refactoring of old contracts for deployment as new service contracts is a vital part of the process. Poorly defined services will not be consumed and will therefore lower the value of the service network.
Certification Kits provide the mechanism for rating the capabilities and therefore the usability of a service. A certified service, provided it meets the well formed service contract requirement, will be trusted and therefore consumed. Services are certified by meeting the requirements of their contract. Obviously this can have various depths.
Certification kits allow service providers not only to guarantee their own kits in terms of meeting the contract but also allow composite/mashup services to gaurantee quality against the services they consume in the mashup. Certification kits include full emulators of service consumer and service provider and include the service contract and the data required for testing. Kits may be black-box or white-box in terms of the type of logic and data required and available. Certification kits require good test data. Some of this can be generated, some can be pulled off of production systems (scrubbed for privacy reasons) and some may need to be hand input. Test data is actually the bulk of the work, not the code.
An accessible, active repository is required to provide a place for efficient storing and retrieval of service contracts, certification kits and certification status. This is not just a look-up registry. When third parties want to consume services, including for consumption within mashups, it is necessary for the process of downloading certification kits to be straightforward. Service network providers may require a fee but the confidence in the certification kits and ease of use will make this worth while. An active repository supports the service lifecycle and will schedule certification tests when new services are deployed into the test environment and can version services across test and production networks. At any point in time consumers and providers can look at the network and see the "health" of the services deployed and therefore the overall "health" of the service network.
This is a very high level look at this model. There are greater details within each of these areas. Feel free to contact me if you are interested in more details.
It's been an exciting month at IONA. First IONA acquired C24 and therefore strengthened both its technology in terms of data services including transformation, but also it's penetration in the financial services industry. Then this week IONA announced availability of a new product Artix™ Registry/Repository which I believe will have a major impact on how companies will view SOA governance.
The C24 acquisition improves IONA's position in the financial services (FS) sector. Artix has been doing quite will in FS because of it's ability to help large FS organizations reuse and interoperate their many IT infrastructure investments. So it's not unusual to see Artix being used to bridge between different technology gaps in trading systems etc. What C24 brings is support for so many of the standard types of message formats that move across these various infrastructure technologies, power transformation engines to support these formats, and of course some customers that have invested in C24 products. Artix Data Services will be a very powerful tool in IONA's SOA story.
Artix™ Registry/Repository is a very powerful innovation for SOAs. I was fortunate to get an early preview of this in Dublin in January and was very impressed with the vision and capabilities. The idea here is that in a distributed SOA infrastructure organizations need a much better handle on design-time, deploy-time and run-time governance. Whereas SOA management technologies, such as Amberpoint, help monitor and manage the flow of messages and data across your SOA, Artix™ Registry/Repository helps insure that the policies on those endpoints are up-to-date and valid. Artix™ Registry/Repository helps designers pull together the policies and configuration of endpoints, and then insure that those policies are maintained. New versions of policies can be pushed to appropriate endpoints and endpoint groupings. Endpoints can notify the repository of policy changes and so central governance can be maintained. So the repository is not just a static collection of metadata but is really the control center for your distributed SOA infrastructure.