Currently, Elastic Beanstalk provides a Java-based application container with Apache Tomcat as the work horse. All you have to do is to deploy a standard Java WAR file containing your web application.

Simply create a Java web application with Spring MVC, Grails, OSGi, Eclipse RAP, or any other of the numerous Java web frameworks and upload it using the AWS web console.

Additional containers for other platforms such as Ruby, Python or PHP may follow later, but as Tomcat hosts standard Java WAR files, anything with a Java implementation may be run. Ruby apps based on Rails or Rack using Warbler and JRuby, Python apps using Jython, even PHP apps can be made to run on Beanstalk.

In this blog post, I’ll dive into the inner workings of a Beanstalk server instance and poke around its internals. I invite you to come along for the ride.

Accessing your instance
The first step in dissecting a Beanstalk instance is getting access via SSH.
In order to acces your running instance(s), you first need to configure the SSH key pair to be used:

Configuration dialog in Elastic Beanstalk web console

Enter the name of a key pair as configured in your EC2 web console. See the EC2 guide and the description in the Elastic Beanstalk guide for details on creating and configuring a key pair.

Setting the key pair requires a restart of your Beanstalk environment, which may take a couple of minutes.

Finally look up your instance id and get the instance’s hostname. Connecting to the server is now as simple as this:

ssh -i .ec2/mykeypair.pem ec2-user@ec2-184-72-134-2.compute-1.amazonaws.com

Please note, that you need to connect as user ec2-user, root access can be reached using the command

sudo su -

Getting around your instance
The first steps are to collect some interessting facts. The instance uses a Amazon Linux AMI (release 2010.11.1 (beta), README). The AMI id for the ElasticBeanstalk-Tomcat6-32bit is ami-7609f81f, the kernel id is aki-407d9529. The instance is EBS-based and there is no ephemeral storage. Currently, Beanstalk is only available in the US East zone.

Software
The process list reveals: along with Apache Tomcat, Beanstalk uses the venerable Apache Web Server. Additional software includes Bluepill for basic process monitoring, and Amazons own HostManager (see below), which is run within a Thin web server.

Network setup
Elastic Beanstalk scales EC2 instances as needed. Therefore the first target is a load balancer provided by Elastic Load Balancing. Each instance runs Apache as the front end on port 80, with web request being reverse proxied into Tomcat on port 8080. Requests for URI /_hostmanager are forwarded to HostManager on port 8999.

CloudWatch performs health checks by periodically requesting the root page (URI /) of your application. Both health check URI and frequency are configurable. If an instance is no longer available or the load changes, CloudWatch starts or stops instances.

[Image from AWS Elastic Beanstalk Concepts blog post]

HostManager
Local instance management is performed by Amazons HostManager. HostManager is a Ruby application based on Rack and running in a Thin server on port 8999. It receives requests on URI /_hostmanager.

Some examples from the access log:

72.21.217.96 (72.21.217.96) - - [21/Jan/2011:22:00:45 +0000] "POST /_hostmanager/tasks HTTP/1.1" 200 368 "-" "AWS ElasticBeanstalk Health Check/1.0"
10.223.61.43 (10.223.61.43) - - [21/Jan/2011:22:01:05 +0000] "GET /_hostmanager/healthcheck HTTP/1.1" 200 90 "-" "ELB-HealthChecker/1.0"
10.122.19.154 (10.122.19.154) - - [21/Jan/2011:22:01:11 +0000] "GET /_hostmanager/healthcheck HTTP/1.1" 200 90 "-" "ELB-HealthChecker/1.0"
10.223.61.43 (10.223.61.43) - - [21/Jan/2011:22:01:36 +0000] "GET /_hostmanager/healthcheck HTTP/1.1" 200 90 "-" "ELB-HealthChecker/1.0"
10.122.19.154 (10.122.19.154) - - [21/Jan/2011:22:01:42 +0000] "GET /_hostmanager/healthcheck HTTP/1.1" 200 90 "-" "ELB-HealthChecker/1.0"
72.21.217.96 (72.21.217.96) - - [21/Jan/2011:22:01:46 +0000] "POST /_hostmanager/tasks HTTP/1.1" 200 368 "-" "AWS ElasticBeanstalk Health Check/1.0"
10.223.61.43 (10.223.61.43) - - [21/Jan/2011:22:02:07 +0000] "GET /_hostmanager/healthcheck HTTP/1.1" 200 90 "-" "ELB-HealthChecker/1.0"
10.122.19.154 (10.122.19.154) - - [21/Jan/2011:22:02:13 +0000] "GET /_hostmanager/healthcheck HTTP/1.1" 200 90 "-" "ELB-HealthChecker/1.0"
10.223.61.43 (10.223.61.43) - - [21/Jan/2011:22:02:38 +0000] "GET /_hostmanager/healthcheck HTTP/1.1" 200 90 "-" "ELB-HealthChecker/1.0"
10.122.19.154 (10.122.19.154) - - [21/Jan/2011:22:02:44 +0000] "GET /_hostmanager/healthcheck HTTP/1.1" 200 90 "-" "ELB-HealthChecker/1.0"
72.21.217.96 (72.21.217.96) - - [21/Jan/2011:22:02:47 +0000] "POST /_hostmanager/tasks HTTP/1.1" 200 368 "-" "AWS ElasticBeanstalk Health Check/1.0"


Files
HostManager is installed in /opt/elasticbeanstalk/srv/hostmanager. A file list can be found here. /opt/elasticbeanstalk/lib also contains a full Ruby 1.9.1 installation.

Some log files can be found at /opt/elasticbeanstalk/var/:

/opt/elasticbeanstalk
/opt/elasticbeanstalk/var
/opt/elasticbeanstalk/var/state
/opt/elasticbeanstalk/var/state/hostmanager.pid
/opt/elasticbeanstalk/var/state/healthcheck
/opt/elasticbeanstalk/var/tmp
/opt/elasticbeanstalk/var/log
/opt/elasticbeanstalk/var/log/LogDirectoryMonitor.log
/opt/elasticbeanstalk/var/log/DaemonManager.log
/opt/elasticbeanstalk/var/log/bluepill.log
/opt/elasticbeanstalk/var/log/hostmanager.log
/opt/elasticbeanstalk/var/log/ApplicationHealthcheck.log
/opt/elasticbeanstalk/var/log/CatalinaLogMonitor.log


Final words
That’s all for today’s blog post. Further posts will take a closer look at HostManager and deployment including the application setup sequence.

I’m looking forward to any feedback and additional information.

Maybe we can even hack HostManager to accept other application containers, e.g. for OSGi applications.

In my professional live I’m working on high availability and monitoring software. When I showed the plugin to my employer,
he asked me to suspend any activities regarding the plugin, until they decide whether this conflicts with our products.

I hope I will be able to continue with the monitor plugin, but until this issue is resolved, I have removed both sources and
downloads.

Regards,

Wolfgang

It’s basically a preview version, but for anybody who was dying to get his or her hands on it, the Grails wiki has all the details on how to install and use the Monitor plugin. For an introduction, see my previous post.

Installation is still a little clumsy as you have to jump through a few hoops because some patches to Grails and the Quartz plugin are needed in order to get the monitor plugin installed and running.

Lots of features are still missing, but for a first version it’s not too bad. I’m open to suggestions, criticism and usability reports. Of course, patches are most welcome

Happy monitoring,

Wolfgang

One aspect of Grails is easy extensibility using plugins. There are already some plugins, which cover a lot of functionality. I have been working on a plugin of my own, a generic monitoring plugin, which will be released shortly under a Apache License.

The monitoring plugin provides pre-defined monitors for many aspects of a web application. Additionally it is really easy to define your own monitors to track usage of application-specific metrics.

The plugin is based around rr4j, which is a Java port of the well-known RRDTool and uses its data storage and graphing engine. It requires the Quartz Plugin (with my patches for GRAILSPLUGINS-190 and GRAILSPLUGINS-213) and Java 1.5 or greater.

So, what does it look like? Some Screenshots (click for bigger image):

So, how do I define a monitor of my own? Following the Grails philosophy of convention over configuration, the monitor plugin defines its own artefact type: Monitor.

Creating an application-specific monitor involves defining a class ending in ‘Monitor’ in the ./grails-app/monitor/ directory and adding some simple elements:

class SessionMonitor implements HttpSessionListener {  def activeSessions = 0L

  static monitorName = 'Sessions'  static monitorGroup = 'Web'  static monitorDescription = """The ${monitorName} monitor contains information regarding active sessions."""

  static monitorDefs = {     'activeSessions'( type:'gauge', aggregation:'avg', min:0) {        description = "Number of active sessions"     }  }

  static monitorGraphDefs = {     'activeSessions'() {        title = '''Active sessions'''        description = title

        // graph sources        source(id:'activeSessions', metric:'activeSessions', aggregation:'average')

        // what to draw        area(id:'activeSessions', legend:true)     }  }

  void sessionCreated(HttpSessionEvent se) {     activeSessions++  }

  void sessionDestroyed(HttpSessionEvent se) {     if (activeSessions > 0) {        activeSessions--     }  }}

The monitor class specifies a name (monitorName), a group it belongs to (monitorGroup, e.g. ‘Web’, ‘System’, or ‘Application’) and optionally a description (monitorDescription). Two DSLs, one for metrics (monitorDefs), one for graphs (monitorGraphDefs), allow to specify the monitored values and what the graphs should look like. Values are collected periodically (currently every minute) using a Quartz-controlled job.

Details will follow once the plugin is released. It still needs some work for the UI and behind the scenes, but a first version will be available soon.

Roadmap

I have plenty of ideas for additional features and monitors.

Features:

JMX export
XML/JSON/CSV export
Support for non-numeric data (e.g. Java version, OS name, hostname, …)
Possibly storage backends other than RRD4J (e.g. a database)
Notification via EMail when certain thresholds are exceeded
Integration of other graphing solutions, like JFreeChart

Monitors:

Count per Domain object
Requests by controller/action
Database/GORM query stats (using Hibernate statistics)
Special monitors for some databases (MySQL “SHOW STATS”, …)
Security monitor for JSecurity/ACEGI/CAS plugins
Monitor for Quartz jobs (job runs, min/max/avg duration)

Basically, the idea is, that every plugin would provide its own set of monitors for the contributed functionality.

I you have any other idea, advice, or feature requests please feel free to post a comment below.