New features arriving in BoxGrinder Build 0.10.0

We are really pleased to announce the next major version of BoxGrinder Build. Version 0.10 includes many exciting new features.

New plugin: libvirt delivery plugin

The most time-consuming feature was the libvirt delivery plugin, enabling appliances to be delivered and registered on the range of platforms that libvirt supports. These include KVM, Xen, VirtualBox and VMWare to name but a few, and whilst the current release of the plugin is a preview, we hope that your feedback will enable us to continue improving it, so please let us know your thoughts!

There are a large number of features and configurable options available, so we shall limit ourselves to some simple examples. We encourage you to consult the documentation for detailed information.

Examples

Deliver the appliance to a qemu hypervisor on the local machine, placing the appliance in the /var/lib/libvirt/images directory, and register the image:

boxgrinder-build definition.appl -d libvirt --delivery-config connection_uri:qemu:///system,image_delivery_uri:/var/lib/libvirt/images  

The plugin will attempt to determine optimal settings by probing the libvirt daemon for its capabilities. If, for instance, the libvirtd residing at the connection_uri machine advertises that it supports kvm, this will be preferred over a slower qemu domain. You can, of course, override these settings manually.

Connect via SSH to a remote vbox (VirtualBox) hypervisor. Deliver the appliance via SFTP to the remote machine and register:

[...] -d libvirt --delivery-config connection_uri:vbox+ssh://root@example.org/session,image_delivery_uri:sftp://root@example.org/var/lib/libvirt/images

In the above example, we assume that you have set up your ssh-agent, thus enabling the plugin to seamlessly use key authentication for both libvirt and SFTP. Password authentication works too, but it requires you to enter a password each time the plugin needs to connect, which is rather inconvenient.

There are a plethora of further features to allow arbitrary modification of domain definitions. Futhermore, we hope to make the plugin more user-friendly for those who require complex configurations, and these are topics we shall blog on in the future.

New plugin: OpenStack delivery plugin

It's a pleasure to submit to testing our OpenStack plugin. The OpenStack plugin allows you to deliver your appliance (in different formats) to a Glance server using its REST API.

Usage of this plugin is very simple. If you want to submit a KVM image to OpenStack:

boxgrinder-build definition.appl -d openstack

...and if you want to have an EC2 image registered:

boxgrinder-build definition.appl -p ec2 -d openstack

Please refer to plugin documentation for detailed usage instructions.

This plugin is in a tech-preview state. This means that we haven't tested it intensively, please help us with some testing. We really appreciate any comments and pull requests for this plugin.

New plugin: VirtualPC platform plugin

If you need the appliance in VHD format which is required by Virtual PC or Citrix XenServer - there you have it. Usage is very simple:

boxgrinder-build definition.appl -p virtualpc

This is an early stage of the plugin as we want to add more functionality in the future like thin/thick disk support.

Kickstart support removal

As of now support for using Kickstart files as input files for BoxGrinder Build is removed. Kickstart files support was the cause of confusion by many BoxGrinder users. They were expecting that the full plugin chain could be executed, but this wasn't ever true. Only operating system plugin execution was tested, everything else was in the hand of gods.

We're now clear about the support: there is no support for kickstart files.

Virtual machines disk alingment

This is not a direct feature of BoxGrinder, but there was a fix commited to the upstream appliance-tools which enables disk alignment by default. From now on, all appliances made by BoxGrinder are aligned.

New supported EC2 regions: sa-east-1 and us-west-2

You're now free to create your AMI's in new regions: sa-east-1 and us-west-2. More info about selecting the regions you can find in S3 plugin documentation.

Enhanced support for variables (parameters) in any string value field of an appliance definition

As requested by the community, it is now possible to use variables in any value field of an appliance:

name: boxgrinder-#OS_NAME#
os:
  name: fedora
  version: 16

Furthermore, you may now inject custom variables from your environment into your appliances simply by defining them.

$ export MY_ENV="I-LOVE-BoxGrinder"

[...]
  base:
    - "mkdir #MY_ENV#" # This becomes "mkdir I-LOVE-BoxGrinder"

There is a hierarchy in which a variable defined in the variables section of an appliance takes precedence over one defined in the environment. See our mailing list post for more detail.

As an added bonus it is now also possible to define nested variables. BoxGrinder can now resolve the following variable #A# successfully as BoxGrinder Rocks!:

#A#: #B# #C#!
#B#: BoxGrinder
#C#: Rocks

You could define these variables either in the variables section of your appliance, or in the environment. The variables section is better for portability, whereas the environment is useful for baking in one-shot customisations such as fixed MAC addresses.

Full release notes can be found below. If you have any comments - find us or our community.

Release Notes

Bug

  • [BGBUILD-152] - Clarification of Post section inheritance behaviour
  • [BGBUILD-242] - Additional EBS overwrite edge cases
  • [BGBUILD-243] - Misleading error messages when YUM mirrors/mirror-lists are unreachable.
  • [BGBUILD-308] - bg-build 0.9.6 complains about file validation based upon file name
  • [BGBUILD-313] - boxgrinder build fails to build ec2 image if ec2-user already exists
  • [BGBUILD-323] - Invalid kernel version recognition makes recreating initrd impossible
  • [BGBUILD-326] - BoxGrinder can fail when build run from /

Enhancement

  • [BGBUILD-304] - Standarize plugin callbacks
  • [BGBUILD-312] - Only use root privileges when necessary
  • [BGBUILD-318] - Add support for us-west-2 region
  • [BGBUILD-320] - support #ARCH# and/or #BASE_ARCH# replacement anywhere in the appliance definition file
  • [BGBUILD-327] - Resolve appliance definition variables in ENV if they are not defined.

Feature Request

  • [BGBUILD-157] - Add Alignment options for virtual appliances
  • [BGBUILD-195] - Add support for OpenStack
  • [BGBUILD-211] - Support for registering appliances with libvirt (KVM/XEN)
  • [BGBUILD-234] - Host contributed appliance definition files
  • [BGBUILD-267] - Add CentOS 6 support
  • [BGBUILD-302] - Add support for VirtualPC platform
  • [BGBUILD-322] - Allow to specify kernel and ramdisk for ec2 plugin
  • [BGBUILD-331] - Add support for sa-east-1 EC2 region

Task

BoxGrinder Build 0.9.8 is out

It has been a long time we met last time! There are a few reasons. We both (Marc and me) were travelling a bit spreading the BoxGrinder love around the world. Marc is still working on a big BoxGrinder improvement - libvirt plugin. Additionally I was pulled into another task - making JBoss AS 7 available in Fedora.

But hey, we found some time to release 0.9.8 :) Although we planned to make 0.9.7 the last release of 0.9.x series - we were forced to do another one. The main reason was that appliance-tools version shipped in Fedora 16 wasn't able to build appliances where GRUB2 is the default boot loader. I took care of this and pushed new appliance-tools version over the weekend, but it required some changes to how we use appliance-creator internally. So, there we have 0.9.8.

CentOS 6 support

As a side effect of fixing appliance-creator - I added the ability to create CentOS 6 appliances. You'll even be able to create CentOS 6 AMIs now! Go, try it and let us know how it went!

Preserving your environment, and limiting time as root

There are some use cases where people get confused about BoxGrinder not using their own environment variables when executing BoxGrinder Build using sudo. Thanks to Marc this is now over, yay! Even more - we'll make sure that the created artifacts have the expected owner, and drop down from root user to standard user (where applicable) as soon as possible. This change also ensures that the user's agents such as ssh-agent are available to BoxGrinder, for instance for the SSH plugin or the upcoming libVirt plugin will seamlessly use your ssh-agent when required. Small things, but makes life easier.

To utilise it at its best, you can simply run boxgrinder-build without sudo. Although even under sudo and su, we now try to behave in a less surprising way.

The release is immediately available in Fedora 15/16 updates-testing repository.

Full release notes you can find below. If you have any comments - find us or our community.

Release Notes

Bug

  • [BGBUILD-310] - BoxGrinder doesnt build appliances when Fedora 16 is the host
  • [BGBUILD-321] - For EBS AMIs use the filesystem type specified for root partition

Enhancement

  • [BGBUILD-312] - Only use root privileges when necessary

Feature Request

BoxGrinder Build 0.9.7 and beyond

A week without BoxGrinder Build release is a lost week :) Let me introduce 0.9.7.

What we broke in the previous release

This small bugfix release fixes some issues regarding the newly introduced files section in 0.9.6.

We also discovered a change in systemd in Fedora 16 which prevented from executing our /etc/rc.local file which lead to unusable AMIs on AWS. The new systemd doesn't have a symlink from /etc/rc.d/rc.local to /etc/rc.local. Even more - a new condition was added to check for the executability for this file that doesn't work when the file is a symlink. See #737047 for more info.

Due to a single character typo swap partitions didn't work for some installations depending on Ruby and operating system combination. Pardon me.

This release is immediately available in the updates-testing Fedora repository.

What we've done to avoid issues in the future

First - issue causes

There are several types of issues that may arrive while developing BoxGrinder Build. Why? BoxGrinder Build isn't just a simple library. We integrate very tightly with operating systems (Fedora, CentOS, RHEL), with our builder (appliance-creator) and with external services (AWS, SSH). If there is a change in some of these - we need to be prepared to handle it. Sometimes upstream introduces a breaking change. For us a change, for example in the file permissions of a single file in an operating system, can break the whole build...

There are of course also ordinary bugs in the code. Nobody is perfect.

What can we do to break things more rarely?

The answer is tests. We have two kinds of tests in BoxGrinder Build: unit tests and integration tests. Unit tests should catch our code bugs, integration tests should catch bugs in the integration.

For our integration tests - we just moved from stable repositories to updates-testing. This way we can be prepared for upcoming changes in the operating systems. We constantly grow our unit tests collection.

We also want to change the way we introduce new features. New features will be added only in major versions, all minor versions will be bugfix only releases. In our case: 0.10.0 is a major version, 0.9.7 is a minor. We may change the version naming in the future to drop the zero in the front, but we'll leave it as is for now.

What to expect after 0.9.7?

We want to stop the 0.9.x series and focus on 0.10.0 now. This means that 0.9.x will not have new releases anymore. Our main area of focus is support for new clouds:

  1. OpenStack - almost done,
  2. CloudStack - tricky, we'll spend some time on it.

We want to build appliances faster using a pre-launched libguestfs service which is theorethically doable.

If you want see our current plans for 0.10.0 - just take a look at our JIRA. As always - our schedule is community-based. If you have some great ideas on how to make BoxGrinder better - JIRA is waiting for you.

Release Notes

Bug

  • [BGBUILD–305] - Incorrect version information in 0.9.6 schema causing validation errors
  • [BGBUILD–307] - Appliance with swap file fails to build if selected OS is centos

Task

  • [BGBUILD–306] - Switch for updates-testing repository for integration tests

BoxGrinder Build 0.9.6 with files support is out

We continue our rapid development and release another minor update to BoxGrinder Build today: 0.9.6.

This release includes mostly bugfixes, but we also have a neat surprise for you, read on.

What's new in 0.9.6?

GRUB fixes

We discovered some small issues that could prevent Xen/KVM/VMware appliances which used GRUB Legacy from running. Fixed!

File section support

Many of you requested an easy way of including selected files into appliances, for example scripts. Our answer thusfar was:

  1. Add your stuff to the post section, or
  2. Create an RPM (which is fairly simple!) and add it to your packages list.

Now this is over! Let me introduce the files section:

name: jeos-f16
summary: fedora 16
os:
  name: fedora
  version: 16
files:
  "/opt":
    - "install_script.sh"
post:
  base:
    - "chmod +x /opt/install_script.sh && /opt/install_script.sh"

This way you have an easy and fast method of including your files into appliances. Feel free to test it and let us know how you like it!

For more information about usage please refer to files section documentation.

Swap support

We hadn't noticed earlier, but there was no way to specify a swap partition in appliance definition files. We have added it now. If you really want to have swap space on your appliance - just specify another partition with swap as the mount point, like this:

hardware:
  partitions:
    "/":
      size: 2
    "swap":
      size: 1

This release will be available in updates-testing Fedora repository soon. That's all for now. You'll hear more from us in the near future, stay tuned :)

Release Notes

Bug

  • [BGBUILD–298] - Fedora 16 or newer has networking issue on platforms different than EC2 because of biosdevname not disabled
  • [BGBUILD–299] - Wrong filenames in GRUB discovery
  • [BGBUILD–301] - Swap feature not working properly

Enhancement

Feature Request

  • [BGBUILD–276] - Import files into appliance via appliance definition file (Files section)

BoxGrinder Build 0.9.5 released

BoxGrinder Build 0.9.5 released

We are pleased to announce the immediate availablity of BoxGrinder Build 0.9.5! This is primarily a bug-fix release, but introduces one major new feature; Fedora 16 build support.

What's new in 0.9.5?

Fedora 16 build support

Fedora 16 has just reached alpha status, and to coincide with this event BoxGrinder Build now provides build support, allowing you to produced Fedora 16 based appliances, including EC2 and EBS based images! BGBUILD–279 BGBUILD–280

This comes with a caveat at present; the current Fedora 16 kernel suffers from the same device ordering problems that occurred with Fedora 15. There is a workaround however, by adding the updates-testing repository into the build:

name: jeos-f16
summary: fedora 16 jeos!
os:
  name: fedora
  version: 16
repos:
  - name: "temp-fedora-updates-testing"
    mirrorlist: "https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f#OS_VERSION#&arch=#BASE_ARCH#"
    ephemeral: true

Once the fixed kernel is in the stable repositories, you won't need this workaround, simply remove the temporary repository entry.

0.9.4 Errata

This release fixes some bugs that were mistakenly introduced into the S3 and EBS plugins during thes rebase of a development branch. The net effect of which was that flaws were created in the release branch that were not present in the tested development branch.

We have improved integration tests to ensure this cannot happen easily again.

Other points of interest

  • BoxGrinder Build will now use more than 4 CPUS (BGBUILD–295), and should run faster than before on machines with larger numbers of cores/processors.

Of CentOS 6, Fedora 15 Kernels, and S3 Overwrite

When will CentOS 6 support arrive

Presently, we are awaiting the resolution of a CentOS 6 bug, and hope to provide build support as soon as the blocker is cleared. You can follow our progress on ticket [BGBUILD-267].

Trouble booting Fedora 15 S3 AMIs

A problematic kernel update (kernel-2.6.40-4.fc15) rendered many Fedora 15 S3 backed AMIs unable to boot, as detailed in RHBZ 729340. The issue has since been rectified, however any Fedora 15 appliances you may have built that used the aforementioned kernel will not function properly, and should be rebuilt. The fixed kernel is in the stable repositories, so simply doing a forced rebuild of any failed Fedora 15 appliances should produce a working machine.

The bug caused device naming to be offset, so whilst Fedora was expecting a root block device named xvda1, it was instead assigned the label xvde1. You should use yum update kernel to ensure you have the latest kernel installed, and if you were unfortunate enough to have an EBS instance rendered unbootable by the bug, then you may be able to revive it with the solution provided by community member jrosengren.

S3 overwrite issues

Particularly observant readers will recall that in the 0.9.4 release blog a bug in aws-sdk library was mentioned, which caused BoxGrinder Build's S3 AMI overwrite feature to fail. Subsequently a new release has been made that fixes the issue, and this has now been packaged and pushed to the Fedora repositories. The issue should no longer apparent in 0.9.5, or in 0.9.4 after a yum update.

Comprehensive Change-log

Bug

  • [BGBUILD–277] - When delivering as AMI, the EC2 region should match S3 bucket’s region
  • [BGBUILD–293] - Check certificate and key paths are valid before building AMIs
  • [BGBUILD–294] - Package aws-sdk 1.1.1 and update dependency
  • [BGBUILD–295] - Remove arbitrary 4 CPU limit
  • [BGBUILD–297] - Cannot create EBS appliances when using overwrite parameter

Enhancement

Feature Request

  • [BGBUILD–255] - Add welcome message for meta appliances
  • [BGBUILD–296] - BG should refer to version and release when building new appliances

Task