One Chef recipe

[https://github.com/thommay/chef-rewind chef-rewind] is a Ruby gem that is needed by most of wrapper cookbooks, as it allows easy overriding of steps inside Chef recipes. A separate recipe for installing chef-rewind should not be needed under normal circumstances, but most of the machines in the LAN don’t have access to https://rubygems.org (which is needed to install external gems). Therefore we have a minimal recipe that copies the gem file to the nodes and installs it locally.

To show how the code works, the easiest thing is to actually go through all the tests for the cookbook and then upload it to the Chef server, making it available to all the nodes (in a specific environment).

The initial step is to cd into the cookbook folder:

xx99pc01:chef$ ls chef-rewind/cookbooks/chef-rewind
 ./  ../  Berksfile  chefignore  files/  .kitchen.yml  metadata.rb  README.md  recipes/

All the elements of a working cookbook are there:

* A Berksfile specifying dependencies from other cookbooks (not very interesting in this case, as this cookbook does not depend on anything)
* A files folder containing our gem archive for chef-rewind
* A .kitchen.yml file describing the setup for integration tests using Test Kitchen
* The metadata.rb manifest providing information about the cookbook
* A README.md to provide a general description in Markdown format
* The recipes folder, only containing the default.rb recipe in our case

We can now install its dependencies using [http://berkshelf.com/ Berkshelf]:

 xx99pc01:chef-rewind$ berks install
 Resolving cookbook dependencies...
 Fetching 'chef-rewind' from source at .
 Fetching cookbook index from https://supermarket.getchef.com...
 Using chef-rewind (0.1.0) from source at .

This step might fail in some cases (giving errors about the proxy and HTTPS URIs). The issue can be solved by changing the command to

xx99pc01:chef-rewind$ https_proxy= berks install

Note: running berks install is not needed if using rake tasks to perform common operations on cookbooks (more on that later).

Using Foodcritic as a lint tool

Now for the tests: an initial sanity check can be done using foodcritic .. (don’t forget the .. argument, as it refers to the enclosing cookbooks directory). The tool checks for syntax errors and style issues in the recipes. Bad practices and anti-patterns will be shown as warnings in the command output; in line with the general behaviour of Unix tools, no output means ‘all good’.

Using rake: From now on, all the commands described will have a shortcut in form of rake tasks. For example, foodcritic can be invoked using the lint task:

xx99pc01:chef-rewind$ rake lint

Please note that the lint task is called automatically by the spec task.

Unit tests with ChefSpec

Next step: testing with ChefSpec. Ensure that all the dependencies are met with bundle. Then execute the unit test:

xx99pc01:chef-rewind$ bundle exec rspec
 chef-rewind::default
 creates the gem file for installing chef-rewind
 installs the chef-rewind gem from the local .gem file
 does not install the gem from a remote repository
 Finished in 0.17926 seconds (files took 1.81 seconds to load)
 3 examples, 0 failures

The unit test might require you to checkout some files in P4:

xx99pc01:chef-rewind$ p4 edit Berksfile.lock
 xx99pc01:chef-rewind$ p4 revert Berksfile.lock

Using rake: the p4:edit task takes care of opening the required files in perforce, while the spec task can be used for running ChefSpec:

xx99pc01:chef-rewind$ rake p4:edit
 xx99pc01:chef-rewind$ rake spec

Let’s have a closer look at the test code (spec/recipes/default_spec.rb):

require_relative ‘../spec_helper’
describe ‘chef-rewind::default’ do
let(:chef_run) { ChefSpec::SoloRunner.new.converge(described_recipe) }
let(:gem_pkg)  { ‘/tmp/chef-rewind-0.0.9.gem’ }
it ‘creates the gem file for installing chef-rewind’ do
expect(chef_run).to create_cookbook_file(gem_pkg)
end
it ‘installs the chef-rewind gem from the local .gem file’ do
expect(chef_run).to install_chef_gem(‘chef-rewind’).with(source: gem_pkg)
end
it ‘does not install the gem from a remote repository’ do
expect(chef_run).to_not install_chef_gem(‘chef-rewind’).with(source: nil)

The test is simply checking that the right actions are invoked by the Chef runtime upon execution of our default recipe. No action is really performed, but we can still check that the recipe is actually trying to do what we expect. A good way of looking at how the recipes are developed is simulating a standard development cycle: first, comment out everything in (recipes/default.rb) and see everything failing:

#cookbook_file “/tmp/chef-rewind-0.0.9.gem” do
#  source “chef-rewind-0.0.9.gem”
#  owner “root”
#  group “root”
#  mode “0644”
#end
#chef_gem “chef-rewind” do
#  source “/tmp/chef-rewind-0.0.9.gem”
#  action :install
#end

xx99pc01:chef-rewind$ rake spec
 [rake] Executing [foodcritic ..] from directory [/home/xx99/com/chef/chef-rewind/cookbooks/chef-rewind]
 [rake] Executing [bundle exec rspec spec] from directory [...]
 chef-rewind::default
 creates the gem file for installing chef-rewind (FAILED - 1)
 installs the chef-rewind gem from the local .gem file (FAILED - 2)
 does not install the gem from a remote repository
 Failures:
 1) chef-rewind::default creates the gem file for installing chef-rewind
 Failure/Error: expect(chef_run).to create_cookbook_file(gem_pkg)
 expected "cookbook_file[/tmp/chef-rewind-0.0.9.gem]" with action :create to be in Chef run. Other cookbook_file resources:
 # ./spec/recipes/default_spec.rb:9:in `block (2 levels) in <top (required)>'
 2) chef-rewind::default installs the chef-rewind gem from the local .gem file
 Failure/Error: expect(chef_run).to install_chef_gem('chef-rewind').with(source: gem_pkg)
 expected "chef_gem[chef-rewind]" with action :install to be in Chef run. Other chef_gem resources:
 # ./spec/recipes/default_spec.rb:13:in `block (2 levels) in <top (required)>'
 Finished in 0.12084 seconds (files took 1.81 seconds to load)
 3 examples, 2 failures
 Failed examples:
 rspec ./spec/recipes/default_spec.rb:8 # chef-rewind::default creates the gem file for installing chef-rewind
 rspec ./spec/recipes/default_spec.rb:12 # chef-rewind::default installs the chef-rewind gem from the local .gem file

Now try to remove the comments around the first block (the cookbook_file resource) and repeat the test:

xx99pc01:chef-rewind$ rake spec
 [rake] Executing [foodcritic ..] from directory [/home/xx99/com/chef/chef-rewind/cookbooks/chef-rewind]
 [rake] Executing [bundle exec rspec spec] from directory [...]
 chef-rewind::default
 creates the gem file for installing chef-rewind
 installs the chef-rewind gem from the local .gem file (FAILED - 1)
 does not install the gem from a remote repository
 Failures:
 1) chef-rewind::default installs the chef-rewind gem from the local .gem file
 Failure/Error: expect(chef_run).to install_chef_gem('chef-rewind').with(source: gem_pkg)
 expected "chef_gem[chef-rewind]" with action :install to be in Chef run. Other chef_gem resources:
 # ./spec/recipes/default_spec.rb:13:in `block (2 levels) in <top (required)>'
 Finished in 0.27105 seconds (files took 1.84 seconds to load)
 3 examples, 1 failure
 Failed examples:
 rspec ./spec/recipes/default_spec.rb:12 # chef-rewind::default installs the chef-rewind gem from the local .gem file

As you can see, everything converges to a ‘green’ state. If we remove all the comments, all tests will pass.

The final test: integration with Test Kitchen

Our coverage is not complete; even in our simple example, are we sure that the recipe we just wrote will work on a real system? ChefSpec only emulates execution, but nothing ‘really’ happens on the developer’s workstation. A nice way to have proof that (under some assumptions) the application of the recipe will work in the real world is to use a virtual machine. The idea here is to spawn a Docker container (as explained earlier) and run chef-client inside it. If our expectations about the final state of the virtual machine are met, we can consider the cookbook ready for production.

As usual, let’s first run the integration tests and see what happens. First of all we can check the status of the virtual images configured in the .kitchen.yml file under the cookbook root:

xx99pc01:chef-rewind$ bundle exec kitchen list
Instance...................  Driver  Provisioner  Last Action
default-ubuntu-1204  Docker  ChefZero     <Not Created>
default-ubuntu-1404  Docker  ChefZero     <Not Created>

As expected, no image created yet. We can generate them with

xx99pc01:chef-rewind$ bundle exec kitchen create
-----> Starting Kitchen (v1.2.1)
-----> Creating <default-ubuntu-1204>...
[some output]
-----> Kitchen is finished. (0m3.45s)

Good, no errors. Now the images will show up as ‘created’:

xx99pc01:chef-rewind$ bundle exec kitchen list
Instance...................  Driver  Provisioner  Last Action
default-ubuntu-1204  Docker  ChefZero     Created
default-ubuntu-1404  Docker  ChefZero     Created

What we can easily do is try to converge them using our default recipe (this is a kind of unit test stored in spec/recipes/default_spec.rb), and manually look for errors:

xx99pc01:chef-rewind$ bundle exec kitchen converge
-----> Starting Kitchen (v1.2.1)
-----> Converging <default-ubuntu-1204>...
[lots of output]
Synchronizing Cookbooks:
- chef-rewind
Compiling Cookbooks...
Recipe: chef-rewind::default
* chef_gem[chef-rewind] action install[2014-12-24T11:22:21+00:00] INFO: Processing chef_gem[chef-rewind] action install (chef-rewind::default line 14)
================================================
Error executing action `install` on resource 'chef_gem[chef-rewind]'
================================================
Gem::Package::FormatError
-------------------------
No such file or directory @ rb_sysopen - /tmp/chef-rewind-0.0.9.gem

This is a typical scenario: an unexpected problem coming from the ‘real world’. Why the file we just copied to /tmp is not there when we install the gem? The reason lies in the order in which the chef_gem statements are processed: since the aim is to provide library code to the actions in the current Chef run, all of the chef_gem resources are processed ”before” the converge phase. Which means we have a problem. Luckily, our cookbook doesn’t ”need” chef-rewind, it just wants to install it to the right location. So let’s modify slightly our recipe:

chef_gem “chef-rewind” do
source “/tmp/chef-rewind-0.0.9.gem”
action :install
end

needs to be changed to

gem_package “chef-rewind” do
source “/tmp/chef-rewind-0.0.9.gem”
gem_binary “/opt/chef/embedded/bin/gem”
action :install
end

xx99pc01:chef-rewind$ bundle exec rspec
 chef-rewind::default
 creates the gem file for installing chef-rewind
 installs the chef-rewind gem from the local .gem file (FAILED - 1)
 does not install the gem from a remote repository
 Failures:
 1) chef-rewind::default installs the chef-rewind gem from the local .gem file
 Failure/Error: expect(chef_run).to install_chef_gem('chef-rewind').with(source: gem_pkg)
 expected "chef_gem[chef-rewind]" with action :install to be in Chef run. Other chef_gem resources:
 # ./spec/recipes/default_spec.rb:13:in `block (2 levels) in <top (required)>'
 Finished in 0.24705 seconds (files took 2.04 seconds to load)
 3 examples, 1 failure
 Failed examples: rspec ./spec/recipes/default_spec.rb:12 # chef-rewind::default installs the chef-rewind gem from the local .gem file

Well, we did change the unit test result with that modification. Of course the expectations in terms of operations performed by the recipe have changed, and therefore we need to change the unit tests as well:

require_relative '../spec_helper'
describe 'chef-rewind::default' do
let(:chef_run) { ChefSpec::SoloRunner.new.converge(described_recipe) }
let(:chef_gem_binary)  { '/opt/chef/embedded/bin/gem' }
let(:gem_pkg) { '/tmp/chef-rewind-0.0.9.gem' }
it 'creates the gem file for installing chef-rewind' do
expect(chef_run).to create_cookbook_file(gem_pkg)
end
it 'installs the chef-rewind gem from the local .gem file' do
expect(chef_run).to install_gem_package('chef-rewind').with(source: gem_pkg, gem_binary: chef_gem_binary)
end
it 'does not install the gem from a remote repository' do
expect(chef_run).to_not install_gem_package('chef-rewind').with(source: nil)

Slightly more complicated, but now all the unit tests are passing again. We can now try to converge the Kitchen VMs again:

xx99pc01:chef-rewind$ bundle exec kitchen converge
-----> Starting Kitchen (v1.2.1)
-----> Converging <default-ubuntu-1204>...
[verbose output]
Chef Client finished, 2/2 resources updated in 4.816504103 seconds
Finished converging <default-ubuntu-1404> (0m9.59s).
-----> Kitchen is finished. (0m18.86s)

All good (finally). But what about automating this process? And we didn’t ”verify” that anything happened, really. That is the job of our last test script (test/integration/default/serverspec/chef_rewind_spec.rb):

require 'serverspec'
# Required by serverspec
set :backend, :exec
describe 'The chef-rewind gem' do
let(:require_statement) { 'require "chef"; require "chef/rewind"' }
it 'is available to the Chef client' do
expect(command("/opt/chef/embedded/bin/ruby -e '#{require_statement}'").exit_status).to eq(0)

Simple and straightforward. Based on the ServerSpec library (just a set of RSpec matchers), the test verifies that the Ruby executable embedded in /opt/chef has access to the gem we just installed. This last step can be performed with the command

xx99pc01:chef-rewind$ bundle exec kitchen verify
-----> Starting Kitchen (v1.2.1)
-----> Verifying <default-ubuntu-1204>...
[other output]
The chef-rewind gem is available to the Chef client

Finished in 3.28 seconds (files took 0.54874 seconds to load)
1 example, 0 failures
Finished verifying <default-ubuntu-1404> (0m15.97s).
-----> Kitchen is finished. (0m35.29s)

The tests passed on both machines (Ubuntu 12.04 and 14.04), as expected.

Using rake: there are shortcuts that can be taken by using rake: to start from a fresh environment (deleting the VM, creating it, converging and verifying it), type

xx99pc01:chef-rewind$ rake cleantest

For a only integration test (after the VM is already created and running), type

xx99pc01:chef-rewind$ rake verify

There are also tasks for executing kitchen subcommands (see rake -T). The kitchen:login command is particularly useful to inspect the changed made with the Chef run.

Upload to the Chef-Server

To perform all the tasks related to cookbook uploading and the general workflow of publishing changes to the Chef server, we will make use of the Berkshelf upload subcommand and [https://github.com/jonlives/knife-spork Knife Spork]. The latter is a knife plugin that helps developers with common tasks, including sensible versioning of cookbooks.

A brief note: specific versions of Chef cookbooks are applied to machines depending on their environments. A [http://docs.chef.io/environments.html Chef environment] is a way to model groups of nodes having a specific function in the organisation. To list the available environments, type

xx99pc01:chef$ rake environments
[] Displaying Chef environment info
[] Executing [knife environment compare production staging development] from directory [/home/xx99/com/chef]
environment: production  staging  development
chef-rewind . . = 0.1.3     = 0.1.3  = 0.1.3
XYZ-aptly . . . . = 0.2.6     = 0.2.6  = 0.2.6
XYZ-nginx . . . = 0.1.2     = 0.1.2  = 0.1.2

As the names suggest, the development, staging and production environment reflect the stability of configurations to be deployed to nodes; the _default environment is needed by Chef, but it will not be used.

Each environment has attributes and cookbook versions. The list of attributes in each environment adjusts the behavior of recipes (e.g. in the case of APT repositories, which are different in production and development), while the versions are used to promote each cookbook from one environment to the next.

To check the integrity of our configuration, the spork commands can be used to check the sanity of the environment.

xx99pc01:chef$ bundle exec knife spork info
xx99pc01:chef$ bundle exec knife spork environment check development

Before uploading, each cookbook should have their version number increased by 1 (bumping). This can be accomplished with rake bump and the general integrity of the cookbook checked with rake check:

[] Executing [bundle exec knife spork check chef-rewind -o chef-rewind/cookbooks] from directory [/home/xx99/com/chef/chef-rewind/cookbooks/chef-rewind]

Checking versions for cookbook chef-rewind... Local Version: 0.1.4 Remote Versions: (* indicates frozen) *0.1.3 *0.1.2 *0.1.1 *0.1.0

At this point we can upload the cookbook and its dependencies. Just type rake upload. Again, in order to ensure the consistency of our deployments, Berkshelf automatically ”freezes” cookbooks after each upload. This means that, once uploaded, a version cannot be changed on the Chef server. Indeed if we try to repeat the operation nothing happens:

xx99pc01:chef-rewind$ rake upload (in /home/xx99/com/chef)
[] Executing [berks install] from directory [...]
Resolving cookbook dependencies...
Fetching 'chef-rewind' from source at .
Using chef-rewind (0.1.3) from source at .
[rake] Executing [berks upload --no-ssl-verify] from directory [/home/xx99/com/chef/chef-rewind/cookbooks/chef-rewind]
Skipping chef-rewind (0.1.3) (frozen)

The last task needed is the ”promotion” of cookbooks, i.e. the association between their latest version with one or more environment. This is easily done with rake:

xx99pc01:chef$ rake promote
OMNI: Promoting chef-rewind
Adding version constraint chef-rewind = 0.1.2
Saving changes to development.json
Uploading development.json to Chef Server
Promotion complete at 2015-01-12 13:32:19 +0000!
Adding version constraint chef-rewind = 0.1.2
Saving changes to staging.json
Uploading staging.json to Chef Server
Promotion complete at 2015-01-12 13:32:19 +0000!

If nothing else is specified on the command line, only the ”development” and ”staging” environments are associated with the new cookbook version; to promote the new cookbook to the production environment, use the command

xx99pc01:chef$ rake promote[production]
Adding version constraint chef-rewind = 0.1.2
Saving changes to production.json
Uploading production.json to Chef Server

Namaste!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s