Inspired by Netflix

by dnurmi

Recently, we (Eucalyptus Systems) held an all-hands event in Santa Barbara where everyone in the company got together to discuss our private/hybrid cloud computing goals, projects, culture, and technology.  We were honored to have Adrian Cockroft present a talk on Netflix’s incredibly impressive open-source cloud platform, which runs in production using Amazon Web Services as the system’s IaaS layer.  From the beginning of the Eucalyptus adventure, we have kept our eyes and ears open, looking for AWS compatible software and systems that should have a ‘just works’ experience when deployed into an AWS compatible Eucalyptus cloud.  Thus, after the presentation, I was immediately enthused to try out some of the subsystems presented on a Eucalyptus cloud, and I decided to chronicle the experience, here.

  • 10am – check out Netflix OSS projects on github
  • 10:02am – choose Edda, which is the project Netflix has created to track AWS entities over time, so that other control systems can maintain/reference a historical record of state transitions
  • 10:03am – read the Edda Wiki and decide to try out the Edda Quick Start Guide
  • 10:04am-11:00am – I learn to java/scala (I’m a C programmer!), make minor Edda code alterations to have it point at a local Eucalyptus 3.2.0 endpoint instead of AWS
  • 11:00am-11:45am – attend non-Edda related meeting
  • 11:46am-noon – walk through the Edda Quick Start guide on github
  • noon-12:15 – Edda basically working against Eucalyptus 3.2.0

Of course, this isn’t a ‘thing’ quite yet, and there are several AWS services that Edda can query/store that are not yet available in Eucalyptus 3.2 (just wait for Eucalyptus 3.3!), but it was exciting to take a real production software system designed and deployed for AWS and see how just how quickly a simple private cloud development/integration environment could be set up using Eucalyptus.

There are a number of other exciting Netflix OSS projects on github just waiting to be similarly pointed at a Eucalyptus private cloud, which will be the subject of upcoming posts!

Running Edda with Eucalyptus

  • Install Eucalyptus 3.2, note the service end-points (EC2_URL and S3_URL in ‘eucarc’)
  • Download Edda and modify ‘./src/main/scala/com/netflix/edda/aws/AwsClient.scala’, replacing all calls to ‘setEndpoint’ with either the ‘/services/Eucalyptus/’ of ‘/services/Walrus/’ endpoints (EC2_URL and S3_URL) for your Euca deployment.  For example:
- client.setEndpoint("ec2." + region + ".amazonaws.com")
+ client.setEndpoint("http://<your Cloud Controller IP address>:8773/services/Eucalyptus/")
  • Follow the Edda Quick Start Guide, using Eucalyptus credentials instead of AWS credentials, and ‘eucalyptus’ as the region (the setting of which is referenced in the Quick Start Guide)
  • Use Edda as documented

For example:

root@gibson:/opt/netflix/edda# curl "$EDDA/view/instances/;_pp"
[
  "i-D8CE40F9",
  "i-828F4452"
]

root@gibson:/opt/netflix/edda# curl "$EDDA/view/instances/i-D8CE40F9;_all;_pp;_meta:(stime,ltime,data:(state:(name)))"
[
 {
  "data" : {
   "state" : {
    "name" : "terminated"
   }
  },
 "ltime" : null,
 "stime" : "2013-01-21T13:04:19.6Z"
 },
 {
  "data" : {
   "state" : {
    "name" : "running"
   }
  },
 "ltime" : "2013-01-21T13:04:19.6Z",
 "stime" : "2013-01-21T12:56:20.2Z"
 }
]

Thus far, I’ve tried addresses, images, volumes, snapshots and reservations which all basically function. Buckets and security groups need some work, and for AWS services that are not available in Eucalyptus 3.2.0, the Edda call returns empty data which is just right.  As we add compatible services, these empty data fields will start populating, which in my opinion is an interesting and meaningful way to measure fidelity between Eucalyptus and AWS.

About these ads