Last week I blogged on Using MRG Grid and EC2. Since then I've been doing some work testing what we've been calling MRG Grid with EC2 Enhanced.
Last week I was submitting an AMI as a job in the cloud. MRG Grid with EC2 Enhanced is different. In this case the user submits an ordinary (vanilla) job but we have a preference to run it in the cloud (EC2). In this case MRG Grid routes the job from the local grid to the cloud. There are some routing rules depending on the characteristics specified for cloud instances. e.g. the job may require the use of high CPU instance types or 64bit based instances.
The job actually causes AMIs (specified in the routing rules) to be launched in the cloud and then uses Web services to transfer data in and out of the running instance.
These rules are in hooks that are fired because of certain job characteristics. There are both pre and post job execution hooks. This allows for the transfer of data in and out.
Obviously the instance(s) running in the cloud need(s) an extra mechanism to get the jobs to run on it's local condor instance. There is a daemon called caroniad that listens for incoming jobs and uses the AWS Web services.
I haven't had to do anything complex yet but so far the MRG / EC2 Enhanced testing has cost be about $4!
As I mentioned in the last blog entry this will all be documented in the MRG 1.1 Grid User Guide. I've been providing some input based on my testing.
Rob Rati is the Red Hat engineer behind caroniad and he's been a great help getting me through the testing.
Last week I was submitting an AMI as a job in the cloud. MRG Grid with EC2 Enhanced is different. In this case the user submits an ordinary (vanilla) job but we have a preference to run it in the cloud (EC2). In this case MRG Grid routes the job from the local grid to the cloud. There are some routing rules depending on the characteristics specified for cloud instances. e.g. the job may require the use of high CPU instance types or 64bit based instances.
The job actually causes AMIs (specified in the routing rules) to be launched in the cloud and then uses Web services to transfer data in and out of the running instance.
These rules are in hooks that are fired because of certain job characteristics. There are both pre and post job execution hooks. This allows for the transfer of data in and out.
Obviously the instance(s) running in the cloud need(s) an extra mechanism to get the jobs to run on it's local condor instance. There is a daemon called caroniad that listens for incoming jobs and uses the AWS Web services.
I haven't had to do anything complex yet but so far the MRG / EC2 Enhanced testing has cost be about $4!
# Example Job Submission for MRG / EC2 Enhanced
universe = vanilla executable = /home/testuser/ec2_tests/my_ec2_job.sh output = ec2test.out$(PROCESS) log = ulog$(PROCESS) requirements = Arch == "INTEL" should_transfer_files = yes when_to_transfer_output = on_exit transfer_executable = true +WantAWS = True +WantArch = "INTEL" +WantCPUs = 1 queue
# More CPU requirements get a large instance type +WantCPUs = 2 queue 2
As I mentioned in the last blog entry this will all be documented in the MRG 1.1 Grid User Guide. I've been providing some input based on my testing.
Rob Rati is the Red Hat engineer behind caroniad and he's been a great help getting me through the testing.
IP Babble is the personal blog of William Henry.