Using MRG Grid and EC2

I've been testing some MRG Grid capabilities recently.  Last week it was using MRG Grid with virtualization and also using MRG Grid with Amazon's EC2.

Using Condor's ability to work with virtualization I was able to bring up a Xen image as a job on my Grid.  This image could have been a complete stack including an application.  I was happy enough just to get an existing RHEL 5.2 guest running. 

Toward the end of the week I was asked to look at our EC2 integration and take it for a test run. My first goal was just to get MRG Grid to launch an image in the cloud.  After I created a new AWS (Amazon Web Services) account  I searched and found a Fedora 6 image already up in EC2 so I decided I'd use that as my first test.  That way I didn't have to go through the task of building my own AMI (Amazon Machine Image).

Next I looked at what changes I'd need to make to MRG Grid's configuration on my laptop - i.e. the local Condor configuration.  The plan was that I'd launch my jobs from my laptop into the cloud.  It turns out that the current MRG Grid configuration (for MRG 1.1 release in December) already has all the default configuration I need in the global configuration file.  So I really had everything that I needed already, in terms of configuration.

Then I had to create a job submission file in order to have MRG Grid's scheduler (Condor's schedd) schedule the job in the EC2 cloud. I used something very similar to the folllowing:

# Note to submit an AMI as a job we need the grid universe
Universe = grid
grid_resource = amazon

# Executable in this context is just a label for the job
Executable  = my_amazon_ec2_job
transfer_executable = false

# Keys provided by AWS
amazon_public_key = cert-ABCDEFGHIJKLMNOPQRSTUVWXYZ.pem
amazon_private_key = pk-AABBCCDDEEFFGGHHIIJJKKLLMMNNOOPP.pem

# The AMI ID
amazon_ami_id = ami-123a456b
amazon_user_data = Hello EC2!

# The keypair file if needed for using ssh etc
amazon_keypair_file = /tmp/keypair
# The security group for the job
amazon_security_groups = MY_SEC_GRP

queue

The key's I got from my AWS account.  The amazon_ami_id is the identifier of an AMI already loaded at EC2. And the security group is only needed if you want to open up certain ports (like 22 for ssh) on the image.  All the values for keys, ids and security groups  above are merely examples.

I then submitted the job and checked using the EC2 command line tools (I had downloaded earlier) and also using Elasticfox - a Firefox plug-in that provides a web condole to view and manage your AMIs.

I did a couple of other steps setting up that security group (above) with the command line tools so that I could ssh into the image. I have to say that running ssh to login to the running image in the cloud made it all seem pretty cool.

I then decided that I would write up my steps and submit them to the MRG Grid User Guide technical writer (God bless her).  So I wrote up the description of the basic steps of using MRG with EC2.  So when MRG 1.1 is released you'll find a more detailed decription of the steps in the MRG Grid User Guide here.

This week I have to work on the "Enhanced" MRG/EC2 integration.  This is where I take advantage of a running AMI in the cloud and use it as a MRG Grid execute node and therefore deploy jobs into it.  Thus making it just another resource available to my Grid, even though it is running externally in the cloud.

Oh, by the way all the EC2 testing on Thursday night and Friday morning cost be about $0.50.  That will be billed to my Amazon account.