Preparing for Fedora 17

Looking to the (beefy) future

Those of you who keep an eye on happenings in Fedora-land will undoubtedly be aware that Fedora 17 is due out in the near future. From BoxGrinder's perspective one of the more important changes is the move to Ruby 1.9.x from 1.8.

There are some syntactic changes and a few subtle semantic differences between versions, so it is important for us to ensure equivalent runtime behaviour in both flavours. Fedora 15 and 16 will both remain on 1.8.7, so we must straddle the fence.

Mercifully, the changes required were fairly minor; but for those of a more inquisitive persuasion, let's look at a few examples of alterations that were required.

Coverage Testing

We needed to provide code coverage analysis with both Rubies, under 1.9 via simplecov and RCov when using 1.8. We only want the relevant tool to be loaded and run for the appropriate version of Ruby.

Our tests are run using Rake, and as Rake tasks run in a new process, a bit of ingenuity is required to ensure the code you write is affecting the correct process.

In this instance the simplest solution was to create a couple of helper files that are run in the new process before the specs begin, ensuring everything required is kicked into action.

Below is a snippet from our Rakefile. Under 1.9, we set an environment variable to indicate to spec_helper that simplecov should be run.

When running with 1.8 a bit of path juggling ensures rcov_helper is run before RCov starts. As RCov is initialised before RSpec, we must ensure that some basic dependencies are met.

Rakefile:'spec:coverage') do |t|
  t.ruby_opts = "-I ../boxgrinder-core/lib"
  t.pattern = "spec/**/*-spec.rb"
  t.rspec_opts = ['-r spec_helper', '-r boxgrinder-core', 
          '-r rubygems', <snip>]
  t.verbose = true

  if RUBY_VERSION =~ /^1.8/
    t.rcov = true
    t.rcov_opts = ["-Ispec:lib spec/rcov_helper.rb", <snip>]
    ENV['COVERAGE'] = 'true'


  require 'simplecov'

  FILTER_DIRS = ['spec']

  SimpleCov.start do
    FILTER_DIRS.each{ |f| add_filter f }

This might seem fairly circuitous, but if we attempted to start code coverage in the Rakefile itself, we'd simply be analysing Rake, not our code!

Sycked up

module BoxGrinder
  # Avoid Psych::SyntaxError (<unknown>): couldn't parse YAML in 1.9
  if RUBY_VERSION.split('.')[1] == '9'
    require 'yaml'
    YAML::ENGINE.yamler = 'syck'

Psych, the default cRuby 1.9.3 YAML parser, causes problems with our YAML parsing and validation (through Kwalify), but fortunately the only change required was to set the parser back to Syck.

Syntactical slips

-    when :ec2:
+    when :ec2
       disk_format = :ami
       container_format = :ami
-    when :vmware:
+    when :vmware
       disk_format = :vmdk

This is a single example of a few slightly unusual case (switch) syntaxes which had crept into the codebase, and due to 1.9's new hash syntax, something like :ec2: appears to be a mangled hash key.

String it out

-    repoquery_output.each do |line|
+    repoquery_output.each_line do |line|

String#each is no longer an alias to #each_line, which splits a string into an array with newline as the separator. The change is probably rather sensible, given that the behaviour is surprising at first encounter.


-    vmdk_image.scan(/^createType="(.*)"\s?$/).to_s.should == "vmfs"
+    vmdk_image.match(/^createType="(.*)"\s?$/)[1].should == "vmfs"
[1] pry(main)> 'createType="BG"'.scan(/^createType="(.*)"\s?$/).to_s
=> "example" # Ruby 1.8.7

[1] pry(main)> 'createType="BG"'.scan(/^createType="(.*)"\s?$/).to_s
=> "[[\"example\"]]" # Ruby 1.9.3

Numerous examples of slightly dodgy regex matching that relied upon to_s in our tests were eliminated from the codebase.

Other bits

Amongst other changes that bit us was our reliance upon quirky behaviour in Ruby 1.8 bindings (albeit our usage is slightly dubious anyway), and a couple of situations where implicit arrays were assumed.

RSpec 2, when dependencies strike

A few of our newer RSpec tests only work properly with rspec-expectations >= 2.7.0, which is available only on Fedora 17 and above.

- it "should add the path to the path_set" do
+ <snip> :if => RSpec::Expectations::Version::STRING >= '2.7.0' do
    expect{ simple_update }.to change(subject, :path_set).

By using RSpec 2 filters we avoid running tests known to fail spuriously. This is a useful technique if you temporarily need to straddle multiple versions, and refactoring isn't desirable.


Thankfully the process was rather easy, with all but a couple of issues being caught by our tests. It would seem that all of the cases were circumstances where we should have been using better approaches anyway, so the outcome was certainly positive.

BoxGrinder Build 0.8.0 features: Using BoxGrinder as a library

This is the second article about the upcoming BoxGrinder Build 0.8.0 features. Previously I highlighted new configuration and CLI. Today I'll show how easy it is to use BoxGrinder from a ruby script.


Currently if you want to use BoxGrinder Build in a script, you are forced to execute a shell process where you specify command line arguments. Although this is a simple solution - it isn't very clean. Take a look at this trivial example:

#!/bin/env ruby
puts "Building appliance XYZ..."
system "boxgrinder build xyz.appl -p vmware -d local"
puts "Done!"

There are a couple of disadvantages to this solution:

  1. Logging - catching STDOUT and STDERR logs from a process and redirecting them to our logger is painful and unreliable. We also lose access to any log levels that are just written to BoxGrinder's log.
  2. Firing up another process, well, sucks.

Work log

Work on this issue was divided in to two tasks:

  1. Generally allow to use it as a library: BGBUILD-79,
  2. Don't require to use file-based appliance definition files: BGBUILD-127.

The first task was quite easy to accomplish. The second was a bit tricky as I wanted to have exactly the same entry point for file- and string-based definitions.


Appliance definition stored in a file

Consider the following:

#!/bin/env ruby
require 'rubygems'
require 'boxgrinder-build'

log = => :trace) "Building appliance XYZ..."

begin'xyz.appl', => :vmware, :deelivery => :local), :log => log).create "Done!"
rescue => e
  log.error e
  log.error "Appliance build failed!"

As you can see we create an Appliance object and execute create method on it. Easy, right?

We also create a Config object where we specify what platform we want to use (:platform), whether we want to remove previous builds (:force) or other parameters. For a full list of parameters, see the Config class source code.

We also inject a logger. You are free to use your own logger of course! The logger must respond to info, warn, error, debug and trace methods. For more info, please take a look at our LogHelper.

Appliance definition stored in an object

If you need to dynamically build the appliance definition, you can pass it as a YAML string instead of writing it to a file:

#!/bin/env ruby
require 'rubygems'
require 'boxgrinder-build'

appliance = {'name' => 'xyz', 'hardware' => {'partitions' => {'/' => {'size' => 5}}}, 'os' => {'name' => 'fedora', 'version' => 14}, 'packages' => ['mc', 'openssh-clients', 'postgresql-server']}

log = => :trace) "Building appliance XYZ..."

begin, => :vmware, :deelivery => :local), :log => log).create "Done!"
rescue => e
  log.error e
  log.error "Appliance build failed!"


The upcoming BoxGrinder Build 0.8.0 will greatly simplify and empower interaction between your Ruby code and BoxGrinder. But if you're not satisfied with the details - feel free to create an issue!