Domain Name System - the forgotten cloud service

Inspired by the good folks at Rackspace (specifically, this post), I realised how important it was to have a easily programmable DNS api for cloud applications and services.

Why do I need to care about DNS ?

Well, its all very well deploying your fantastic app to a fantastic elastic cluster, but its not so fun when it has urls like

http://somethingwierd.4324324.xyz.zw.com
http://203.143.2.23

(I hope I haven't offended anyone by using those addresses - there was one time Van Halen put a serial number on the back of an album that happened to be someones phone number, so crazed fans called the number, over and over, until Van Halen was sued over it. But I digress). So to avoid being sued due to over zealous Van Halen fans, you need to have a naming service: DNS. The fabric of the internet, full of quaint terms like "bailiwick" (I wish I wasn't making this up). For single apps, you can use either your own DNS server(s), or use a pay for service etc (there are many options of course). Running your own DNS is nice, it gives you flexibility to run many apps under one domain (using subdomains for specific apps), use it to "move" apps around whenever you need (which is pretty common in clouds - servers can come and go, change data centres, change IPs as you bring up a new one etc).

A simple API

Wouldn't it be nice to have some near API to make it as simple as:

curl -d subdomain=applicationName&address=1.2.3.4 http://coolingtower/api/naming/domains/samplezone.org
curl -d subdomain=applicationName&address=somewhere.com http://coolingtower/api/naming/domains/samplezone.org

Well you would be in luck! (It is a nice rule of thumb that a RESTful interface can be driven from the command line - the above using the excellent curl tool - you can use anything of course). However, DNS is full of traps for young players, and really, what we want is to be a big Map<String, IP> in the sky, magically guiding browsers to the right server.

Where and when would I use this?

Mainly for the entry points that end users will see in their browser- but it can also be useful for internal resources like databases, and other services that your applications consume (to avoid binding to IPs directly). For example, imagine you have your own cloud service, you register name for the top level domain: stormgrind.com. With a DNS service you can have:

http://demo.stormgrind.com
http://db1.stormgrind.com
http://www.stormgrind.com

...and so on, you get the idea.

A RESTful API for DNS management

CoolingTower has a programmatic (RESTful) interface to manage DNS mappings, and also an (optional) authoritative DNS server (if you don't have one already). DNS data is stored in standard "zone files". It will validate and store the changes in zone files - and those changes can be propagated to the DNS servers. If you use the EagleDNS service it will instantly pick up the changes and apply them - applying sensible defaults, updating serial numbers, setting short time to live values (so changes are applied quickly) - all are done automatically to keep things easy and less error prone. There is much more to the RESTful api, and it is being documented.

More info and some links

  • CoolingTower naming service documentation/wiki.
  • Build/source install instructions here.
  • EagleDNS is can be obtained from here (the latest release has my patches for automatically updating zones are they are changed).

Red Hat Open Source Cloud Computing Forum

Red Hat is hosting its 2nd Open Source Cloud Computing Forum on 10 February. This is a free, online event, so prepare some time on 10th February. We'll have two sessions there:

  • BoxGrinder on 10:00 AM, 10 Feb 2010 - Bob McWhirter
  • Cooling Tower on 3:30 AM, 10 Feb 2010 - Michael Neale

We're happy to meet you there!

RHQ CLI – configuring and importing resources

RHQ is the underlying platform for all management projects from JBoss: commercial supported JBoss Operation Network and JOPR – a community based monitoring and managing suite.

Some time ago RHQ developers announced new RHQ build including all JOPR plugins. After that we decided to move from JOPR to RHQ and use latest developer build.

RHQ/JOPR in the Cloud

CirrAS goal is to deploy a cluster of JBoss AS without user interaction at all. Now, we're expanding the goal and adding a monitoring and management (still no user actions needed!).

On each node we're installing an RHQ Agent. To be honest it is installed automatically after the node (front-end, back-end) is started. Every node grabs plugin binary from RHQ Server. The binary is installed then and finally a RHQ Agent is launched.

An agent is responsible for gathering information about running services and reporting that to RHQ server which is located on management appliance. A discovered node is added to Discovery Queue in RHQ Server. This was a scratch on glass – earlier an administrator was forced to log in into RHQ console and import selected resources. This is now over!

Second thing we wanted to avoid is to configure any plugin to have a working monitoring suite. Unfortunately Apache HTTP Server plugin requires an URL to put into configuration. This URL is used for checking availability. We worked it out too using RHQ CLI.

RHQ CLI

RHQ has a powerful Command Line Interface included. Before you're able to use it, you need to install it properly (not a big deal). If you think RHQ CLI language looks familiar, yes, you're right – it's JavaScript, but extended to use also Java objects.

There are two ways you can use this CLI: interactive and non-interactive. In interactive mode you have a Bash-like command line where you execute commands. Greg Hinkle described it very well. In non interactive mode all commands located in a file are executed. Your choice.

Before we start I need to mention that JavaDocs for RHQ Remote API and domain are very helpful!

Let's begin! First of all we want to grab all discovered but not imported Apache HTTP servers.

var criteria = new ResourceCriteria();
criteria.addFilterResourceTypeName("Apache HTTP Server");
criteria.addFilterInventoryStatus(InventoryStatus.NEW);
var httpd_servers = ResourceManager.findResourcesByCriteria(criteria);

A list of resources is returned, we can print all elements:

if (httpd_servers != null && httpd_servers.size() > 0) {
    for (var i = 0; i < httpd_servers.size(); i++) {
        var resource = httpd_servers.get(i);
        println(" - " + resource.name);
    }
} else {
    println("No servers found.")
}

Making changes in plugin configuration

We want to get the configuration for agent connection which we want to modify. We need to grab a proxy object for selected resource. Proxy objects are simplifying interaction with RHQ resources.

var httpd = ProxyFactory.getResource(httpd_servers.get(0).id);

From that object we need to grab plugin configuration which we need mot alter:

var httpd_configuration = httpd.getPluginConfiguration();

Last step is to simply modify property we want:

httpd_configuration.getSimple("url").setStringValue("http://10.1.0.3");
httpd.updatePluginConfiguration(httpd_configuration);

Plugin configuration is now modified. Simple, huh?

Importing resources

Next we want to import resources we discovered earlier. It is as simple as executing importResources function:

DiscoveryBoss.importResources(httpdResourceIds);

Please be aware that this command takes one argument – an array of resource ids. You'll get also an error when you try to import a resource with a parent resource which is not imported. You need to import resources in the proper order, remember! You can use for that function like this:

httpdResourceIds = [];

for (i = 0; i < httpd_servers.size(); i++) {
    addDependencyIds(httpd_servers.get(i), httpdResourceIds);
}

function addDependencyIds(resource, array) {
    var parent = ResourceManager.getResource(resource.id).getParentResource();

    if (parent != null) {
        addDependencyIds(parent, array);
    }

    array.push(resource.id);
}

Conclusion

RHQ CLI is really great. Above example is only a tip of the iceberg what you can do with it. In CirrAS we're using a bigger script which solved our problem with auto importing and reconfiguring servers. You can take a look, maybe it'll help you with your problem too!

Screencast: Get started with CirrAS on Amazon EC2

Our first screencast shows setting up a JBoss AS 6 cluster using StormGrind CirrAS on Amazon EC2. More info on CirrAS you can find on project page and documentation wiki. Be sure also to read our previous announcement.

View this screencast in HD on Vimeo.

If you're interested in music in the background, go grab it, it's free. Downloadable versions:

Announcing StormGrind (and new releases!)

Welcome to our new blog!

Introducing StormGrind Blog

This is a successor of oddthesis.org blog. Here you'll find articles and announcement of new releases for all StormGrind projects. P.S. All cloud related articles located at oddthesis.org will be migrated shortly, so don't be afraid :)

Introducing StormGrind

StormGrind located on JBoss community pages is an umbrella for cloud projects found in JBoss. If you have questions about the project read first our FAQ document located on our new Wiki. We'll consequently update this document to answer all your big questions. Currently under StormGrind we have 7 projects:

  • Cantiere -- RPM building helper,
  • BoxGrinder -- Set of appliance tools;
    • BoxGrinder Build -- tool for building appliances
    • BoxGrinder REST -- REST interface to BoxGrinder Build
    • BoxGrinder Studio – web interface to BoxGrinder Build (no website yet – in planning),
  • CirrAS -- JBoss AS clustering in the Cloud.

A few of above projects are new, others are refactored, enhanced and renamed projects from oddthesis.org incubator. More information about name changes you can find on our wiki. Source code was also migrated to new location. We're still on GitHub – check out stormgrind user! More info on source code on our project site. If you're still confused with name transition feel free to ask us on IRC, see below for more information.

Community and contact with developers

We currently don't have new mailing list. We're in the process of setting up forums for you. Forum availability will be announced in a separate post. If you're using IRC, we're ready for a talk in #stormgrind room on freenode.net. We have of course a shiny new Twitter account, follow us! Always up-to-date information about community you can find on our community page. Our new issue trackers are ready for your input!

New releases

We're also happy to announce first versions of CirrAS, BoxGrinder Build and Cantiere.

BoxGrinder Build 1.0.0.Beta1 released!

First Beta version of BoxGrinder Build is out now! BoxGrinder Build is a project formerly know as JBoss Appliance Support – a set of Rake tasks to build appliances from simple plain text appliance definitions. You can find more information on BoxGrinder Build on project page and documentation wiki.

1.0.0.Beta1 highlights

With the move to BoxGrinder from JBoss Appliance Support, appliance definition files structure has changed. Now you don't need to specify OS name and version (and many more!) via command line parameters, everything now is in one place – in appliance definition file. Read more about new appliance definition file structure on our wiki. We removed wizard feature in this version. Wizard was good when we had many command line parameters, but now we're using simple plain text appliance definition files and every information you're earlier specifying in command line parameters you can now put conveniently in this file.

Download

Release notes for Beta1 can be found here and release is available for immediately download on the download page.

CirrAS 1.0.0.Beta1 released!

A first version of CirrAS – 1.0.0.Beta1 is also out! CirrAS is an effort make deploying of JBoss AS cluster in the Cloud as smooth as possible. More information is available on CirrAS project page and in the wiki.

1.0.0.Beta1 highlights

CirrAS is JBoss AS 6.0.0.M1 based! (JBoss Cloud was using JBoss AS 5). This helped to fix CIRRAS-6. CirrAS has a standalone JOPR installation for monitoring and managing services. The cool thing is that every discovered node is registered automatically. JOPR itself is automatically deployed on management appliance without the need to run configuration wizard, etc. To access JOPR console just point your browser to management appliance address and port 7080. Username and password are default for JOPR installations: rhqadmin/rhqadmin.

Download

Summary of work done is available here. You can download CirrAS bits from CirrAS download page.

Cantiere 1.0.0.Beta1 released!

All RPM building stuff from previous JBoss Appliance Support is now moved to Cantiere. It is a helper tool for building RPM files from spec files. For more information about using Cantiere refer to our wiki.

Download

You can download Cantiere from download page.