IPBabble

Last week I blogged on Using MRG Grid and EC2.  Since then I've been doing some work testing what we've been calling MRG Grid with EC2 Enhanced. 

Last week I was submitting an AMI as a job in the cloud.  MRG Grid with EC2 Enhanced is different. In this case the user submits an ordinary (vanilla) job but we have a preference to run it in the cloud (EC2).  In this case MRG Grid routes the job from the local grid to the cloud.  There are some routing rules depending on the characteristics specified for cloud instances. e.g. the job may require the use of high CPU instance types or 64bit based instances.

The job actually causes AMIs (specified in the routing rules) to be launched in the cloud and then uses Web services to transfer data in and out of the running instance.

These rules are in hooks that are fired because of certain job characteristics. There are both pre and post job execution hooks.  This allows for the transfer of data in and out.

Obviously the instance(s) running in the cloud need(s) an extra mechanism to get the jobs to run on it's local condor instance.  There is a daemon called caroniad that listens for incoming jobs and uses the AWS Web services.

I haven't had to do anything complex yet but so far the MRG / EC2 Enhanced testing has cost be about $4!

# Example Job Submission for MRG / EC2 Enhanced
universe = vanilla
executable = /home/testuser/ec2_tests/my_ec2_job.sh
output = ec2test.out$(PROCESS)
log = ulog$(PROCESS)
requirements = Arch == "INTEL"
should_transfer_files = yes
when_to_transfer_output = on_exit
transfer_executable = true
+WantAWS = True
+WantArch = "INTEL"
+WantCPUs = 1
queue 

# More CPU requirements get a large instance type
+WantCPUs = 2
queue 2

As I mentioned in the last blog entry this will all be documented in the MRG 1.1 Grid User Guide.  I've been providing some input based on my testing.

Rob Rati is the Red Hat engineer behind caroniad and he's been a great help getting me through the testing.

I've been testing some MRG Grid capabilities recently.  Last week it was using MRG Grid with virtualization and also using MRG Grid with Amazon's EC2.

Using Condor's ability to work with virtualization I was able to bring up a Xen image as a job on my Grid.  This image could have been a complete stack including an application.  I was happy enough just to get an existing RHEL 5.2 guest running. 

Toward the end of the week I was asked to look at our EC2 integration and take it for a test run. My first goal was just to get MRG Grid to launch an image in the cloud.  After I created a new AWS (Amazon Web Services) account  I searched and found a Fedora 6 image already up in EC2 so I decided I'd use that as my first test.  That way I didn't have to go through the task of building my own AMI (Amazon Machine Image).

Next I looked at what changes I'd need to make to MRG Grid's configuration on my laptop - i.e. the local Condor configuration.  The plan was that I'd launch my jobs from my laptop into the cloud.  It turns out that the current MRG Grid configuration (for MRG 1.1 release in December) already has all the default configuration I need in the global configuration file.  So I really had everything that I needed already, in terms of configuration.

Then I had to create a job submission file in order to have MRG Grid's scheduler (Condor's schedd) schedule the job in the EC2 cloud. I used something very similar to the folllowing:

# Note to submit an AMI as a job we need the grid universe
Universe = grid
grid_resource = amazon

# Executable in this context is just a label for the job
Executable  = my_amazon_ec2_job
transfer_executable = false

# Keys provided by AWS
amazon_public_key = cert-ABCDEFGHIJKLMNOPQRSTUVWXYZ.pem
amazon_private_key = pk-AABBCCDDEEFFGGHHIIJJKKLLMMNNOOPP.pem

# The AMI ID
amazon_ami_id = ami-123a456b
amazon_user_data = Hello EC2!

# The keypair file if needed for using ssh etc
amazon_keypair_file = /tmp/keypair
# The security group for the job
amazon_security_groups = MY_SEC_GRP

queue

The key's I got from my AWS account.  The amazon_ami_id is the identifier of an AMI already loaded at EC2. And the security group is only needed if you want to open up certain ports (like 22 for ssh) on the image.  All the values for keys, ids and security groups  above are merely examples.

I then submitted the job and checked using the EC2 command line tools (I had downloaded earlier) and also using Elasticfox - a Firefox plug-in that provides a web condole to view and manage your AMIs.

I did a couple of other steps setting up that security group (above) with the command line tools so that I could ssh into the image. I have to say that running ssh to login to the running image in the cloud made it all seem pretty cool.

I then decided that I would write up my steps and submit them to the MRG Grid User Guide technical writer (God bless her).  So I wrote up the description of the basic steps of using MRG with EC2.  So when MRG 1.1 is released you'll find a more detailed decription of the steps in the MRG Grid User Guide here.

This week I have to work on the "Enhanced" MRG/EC2 integration.  This is where I take advantage of a running AMI in the cloud and use it as a MRG Grid execute node and therefore deploy jobs into it.  Thus making it just another resource available to my Grid, even though it is running externally in the cloud.

Oh, by the way all the EC2 testing on Thursday night and Friday morning cost be about $0.50.  That will be billed to my Amazon account. 

There is a great interview with Jim Whitehurst on ZDNet yesterday that starts on the topic of Cloud computing and virtualization and ends up speaking about the merits of open source.  Jim's ability to explain the value of these innovations is really clear and to the point. 

On the topic of Cloud, Jim explains that he believes that the economics of server farms will win out.  He then says "whether they are internal private grids or clouds, or whether those are public, or semi-private, that's still to be determined, because that's a lot around specific business models."  Therefore flexibility to use private grids or clouds or external clouds will be come very important for many companies.  This flexibility is one of the Objectives of MRG Grid.

Jim then does a great job of explaining how virtualization is becoming a commodity and should be moved to the operating system layer.  Then he explains the Red Hat business model for open source.  Again, he makes it very concise and clear.  Worth a read.

My blog difficulties have delayed this blog entry. I wanted to do a "Farewell IONA" blog (and I might do something later) and then an introduction to my new job at Red Hat.

After over 11 years at IONA (my couple of years at Aurora Technologies are officially included) I moved to Red Hat in May of this year.  Not long after my move I started hearing rumors of IONA's acquistion. And, of course, now IONA is part of Progress Software.

My move was prompted much less by IONA's fate than by what Red Hat is doing with MRG. It was time for change and MRG represents something very exciting going on in the industry. There is a lot I can and will be saying about MRG on this IPBabble and I'll only give a short introduction in this entry.

Red Hat's expansion into emerging technologies is an effort to provide more value to its customers. This work includes, among other things, virtualization, management and Messaging, Realtime and Grid (MRG).  I'll be explaining more about details in later posts but briefly:

• Messaging - an open source (Qpid), open standard (AMQP) based, platform independent, messaging infrastructure.
• Realtime - a predictable, low-latency linux kernel.
• Grid - an open source (Condor) HPC, HTC platform that provides the flexibility of local grids, remotes grids and cloud computing.

Of course there are lots more details but that is the very high level view. I'll be diving in a lot more on ipbabble.com.

I have a hybrid role: I work for Brian Stevens, the CTO, in the Office of CTO focused on MRG. In that way I'm very much an evangelist and customer focused solution architect. But in the same organization I'm an engineer working on MRG. So I have to contribute to the engineering effort too. It is both a very rewarding and very challenging role.  Getting up to speed on all the various aspects of MRG has been a steep learning curve despite my background in distributed computing and messaging. Thankfully I work with a really smart and dedicated team that has been generous with sharing their knowledge.  I'm sure I'll be introducing some of them on this blog in the future.


UPDATE: Thank you to Lana for spotting the "bog" typo.

Well I have this new blog set up but I can't figure out how to move all my old blog entries into this blog.  I guess I didn't export them correctly and just zipped up the files. If anyone knows how to get old Movable Type based blog entries into a new version of MT then let me know.

Update:
I'm just loading these manually and setting the date to the original published date.  I doubt I'll get around to entering the comments and trackbacks for the old entries. I'll have to see.

I have moved IPBabble.com's hosting service and I'm trying to move all my previous content over.

Thank you for your patience.