William Liu

Chef

Summary

Chef is a configuration management tool written in Ruby and Erlang. It uses a domain-specific language (DSL) for writing system configuration recipes. Chef helps solve the problem of configuring and maintaining (configure/manage) a company’s servers. Chef utilizes a declarative approach, meaning we specify what the final configuration should be and not have to specify exactly all the steps needed to make it happen.

Infrastructure As Code

Chef falls under the whole Infrastructure as Code idea. Benefits include:

Chef Components

Chef Infra is a systems and cloud infrastructure automation framework. Each organization is made up of one or more ChefDK Installations, a single Chef server, and every node that will be configured and maintained by Chef Client. Cookbooks (and recipes) are used to tell Chef Client how to do the actual configuration.

Tools

Some important tools in ChefDK are:

Desired State

Chef cares about the end result, not the steps to get there.

So what does this all mean? There’s two ways to define our configuration, either Imperative vs Declarative.

Imperative Configuration

An imperative configuration might look like:

yum install -y nginx
systemctl start nginx
systemctl enable nginx

We have to be very specific about what happens.

Declarative Configuration (using Chef DSL)

A declarative configuration might look like:

package "nginx"

service "nginx" do
  action [:enable, :start]
end

The above doesn’t care if we’re running systemd.

Chef Workflow

ChefDK defines a common workflow for cookbook development as:

  1. Create a skeleton cookbook - basically a cookbook with the standard files already included The package manager is usually Berkshelf, which is part of the ChefDK. We also install a revision control system (usually Git). Berkshelf helps manage cookbooks and cookbook dependencies
  2. Create a virtual machine environment using Test Kitchen. This is the environment used to develop the cookbook, including the location that does automated testing and debugging of what cookbook will be done as it is being developed
  3. Write the recipes for the cookbook and debug those recipes as they are being written
  4. Perform acceptance tests on a full Chef Server (not a local development environment) that mimics a production environment as much as possible
  5. When the cookbooks pass acceptance tests and have been verified to work in the correct mannger, deploy the cookbooks to the production environment

Installation

Install Chef Server

  1. Download chef-server-core
  2. Install chef-server-core (e.g. rpm -Uvh chef-server-core.rpm) to install Chef Server and provide a few utilities
  3. Run chef-server-ctl reconfigure to configure its own services
  4. Run chef-server-ctl service-list to see what services the chef server manages E.g. okshelf nginx oc_bifrost oc_id opscode-chef-mover opscode-erchef postgresql
  5. Create a user chef-server-ctl user-create USER_NAME FIRST_NAME LAST_NAME EMAIL 'PASSWORD' --filename FILE_NAME E.g. chef-server-ctl user-create will Will Liu william.q.liu@gmail.com 'mypassword' --filename /home/user/will.pem where we will output an RSA key used to interact with the Chef server from a workstation later on
  6. Create an organization chef-server-ctl org-create SHORT_ORG_NAME 'FULL_ORG_NAME' --association_user USER_NAME --filename FILE_NAME E.g. chef-server-ctl org-create myorg 'My Organization' --association_user will --filename my-validator.pem where the –association_user flag takes an existing user's username and associates it with the admin security group The –filename` flag stores the organization’s validator pem
  7. Optional: Install chef-manage web ui (an add-on that gives a web-based way to see all the chef related info regarding our node information, cookbook versions) using chef-server-ctl install chef-manage Then run chef-server-ctl reconfigure and then chef-manage-ctl reconfigure

Install ChefDK

ChefDK is the development kit that we will install on our workstation (e.g. our labtop)

  1. Download chefdk.rpm
  2. Install rpm -Uvh chefdk.rpm
  3. Now we have access to different tools for developing and testing our Chef code; check version with chef --version chef –version Chef Development Kit Version: 2.5.3 chef-client version: 13.8.5 delivery version: master (73ebb72a6c42b3d2ff5370c476be800fee7e5427) berks version: 6.3.1 kitchen version: 1.20.0 inspec version: 1.51.21
  4. Make sure you have the right ruby which ruby
  5. Use the Chef specific version of ruby eval "$(chef shell-init bash)"
  6. Generate a chef-repo that will hold our cookbooks and dependencies among the team with chef generate repo generated-chef-repo When we use a chef generator, it makes it easier to follow standard Chef development practices
  7. The main tool we will use to connect to the Chef server is the knife utility. Configure it with knife configure
  8. Run knife commands (e.g. knife node list, knife ssl fetch)
  9. Optional: Instead of setting up a chef-repo, we can also download a chef-starter kit

Install Chef Node

A Chef Node is the server that we will configure using chef. We won’t ssh into this server and will basically just run knife commands on it.

  1. Create a server (e.g. say its at myserverlocation.someplace.com)
  2. On your workstation, run knife with -x (for ssh USERNAME), -N (pass in a node name), -P (for password) In our chef-repo, we will run knife bootstrap myserverlocation.someplace.com -N web-node-1 -x user -P 'mypassword' --sudo
  3. This runs a few steps, including creating a new client for ‘web-node-1’
  4. Now that a node is registered, we can run knife node list and you should see web-node-1
  5. What this does is that Chef uses a pull-based approach for configuration changes using chef-client and we’ve now setup chef-client to run on our ‘web-node-1’

The chef-client process:

  1. Get configuration data - read information from client.rb file and Ohai attributes
  2. Authenticate w/ Chef server - Uses RSA key and node name to authenticate with Chef server. Will generate a new RSA key if this is the first connection
  3. Get/rebuild the node object - Pull node object from Chef server if this isn’t the first chef-client run. After the pull, the node object is rebuilt based on the node’s current state.
  4. Expand the run-list - compiles the list of roles and recipes to be applied
  5. Synchronize cookbooks - Request and download all of the files from cookbooks on the Chef server that are necessary to converge the run list and are different from the files already existing on the node
  6. Reset node attributes - rebuild the attributes on the node object
  7. Compile the resource collection - load the necessary ruby code to converge the run-list
  8. Converge the node - execute the run-list
  9. Update the node object, process exception & report handlers - Update the node object on the Chef server after the chef-client run finishes successfully. Also execute the exception and report handlers in the proper order.
  10. Stop, wait for the next run - the chef-client waits until the next time it is executed

Supermarket

The Chef Supermarket is basically like the pypy, rubygems, or dockerhub, except we have a lot of cookbooks and plugins. Even though there is a public supermarket, you can run the supermarket code and deploy a private supermarket.

There is Berkshelf (kinda like Ruby’s Bundler) to install and manage cookbook dependencies and versions (aka dependency manager) There is also Stove used to version and publish cookbooks to a supermarket (either public or private)

Test Kitchen

We use test kitchen (aka Kitchen) as a testing harness that allows us to easily provision environments using a variety of platforms and backends so that we can test our infrastructure code. The basic structure for a kitchen.yml file looks like:

driver:
  name: driver_name

provisioner:
  name: provisioner_name

verifier:
  name: verifier_name

transport:
  name: transport_name

platforms:
  - name: platform-version
    driver:
      name: driver_name
  - name: platform-version

suites:
  - name: suite_name
    run_list:
      - recipe[cookbook_name::recipe_name]
    attributes: { foo: "bar" }
    excludes:
      - platform-version
  - name: suite_name
    driver:
      name: driver_name
    run_list:
      - recipe[cookbook_name::recipe_name]
    attributes: { foo: "bar" }
    includes:
      - platform-version

DNS Issues

A lot of times you’ll run into DNS / Network issues. You can usually add -V to make your command more verbose to get more feedback. You can modify the /etc/hosts if you want to connect to servers without needing DNS to propogate.

Cookbooks

Chef Cookbooks are the unit of configuration and policy distribution. A cookbook defines a scenario and contains everything that is required to support that scenario. You can only have one cookbook of a specific name. Use underscore instead of dashes in names.

Attributes

An attribute is a specific detail about a node. Attributes are used by Chef Infra Client to understand:

Attribute Types

There are six types of attributes to determine the value that is applied to a node during a Chef Infra Client run. Most of the time, we just use default

default - a default attribute is automatically reset at the start of every Chef Infra Client run and has the lowest attribute precedence. Use default as often as possible in cookbookes.

Other attribute types include: force_default, normal, override, force_override, automatic

Files

Files are managed using the following resources:

Libraries

A library allows arbitrary Ruby code to be included in a cookbook, usually included in the /libraries directory. You can do anything in a library file. Good use cases include:

Recipes

A recipe is the most fundamental configuration element within the organization. A recipe is:

Apply to Run-lists

A recipe has to be assigned to a run-list using the appropriate name (as defined by the cookbook directory and namespace). For example, with the following structure:

cookbooks/
  apache2/
    recipes
      default.rb
      mod_ssl.rb

We have two recipes (`default.rb` and `mod_ssl.rb`), which if it translated to a run-list, would look like:

{
  'run_list': [
    'recipe[cookbook_name::default_recipe]',
    'recipe[cookbook_name::recipe_name]'
  ]
}

The `default_recipe` does not need to be specified since it is implied.

Resources

A resource is a statement of configuration policy that:

Resource Syntax

A resource is a Ruby block with four components: a type, a name, one (or more) properties (with values), and one (or more) actions. The syntax looks like:

type 'name' do
  attribute 'value'
  action :type_of_action
end

Every resource has its own set of actions and properties. Depending on the resource type, there is always a default action (e.g. package resource has the default action :install and the name of the package defaults to the name of the resource)

For example, a resource that is used to install a tar.gz package for version 1.16.1 looks like:

package 'tar' do
  version '1.16.1'
  action :install
end

This can also be written as:

package 'tar'

The resource block that installs a tar.gz package would look like this since we do not have to specify the default action of install.

package 'tar' do
  version '1.16.1'
end
Resource Actions

https://docs.chef.io/resource_common.html

You can specify a lot of different actions for your resource, including :nothing (which does nothing). This looks like:

service 'memcached' do
  action :nothing
end
Resource Properties

https://docs.chef.io/resource_common.html

You can specify a lot of different properties for your resource. Some more common properties that are common to every resource include:

An example of using one of these properties looks like:

service 'apache' do
  action [ :enable, :start ]
  retries 3
end

Guards

Additional Resource Properties include guards, which is evaluated during the execution phase and tells Chef Infra Client whether or not it should continue executing a resource. A guard accepts either a string value or a Ruby block value and has a prefix of not_if or only_if.

execute 'bundle install' do
  cwd '/myapp'
  not_if 'bundle check' # this is run from /myapp
end

Lazy Evaluation

In certain cases the value for a property cannot be known until the execution phase of a Chef Infra Client run. Use a lazy evaluation looks like this:

attribute_name lazy { code_block }

Notifies

A resource can notify another resource to take action when its state changes.

notifies :action, 'resource[name]', :timer

An example might look like:

template '/etc/nagios3/configures-nagios.conf' do
  # other parameters
  notifies :run, 'execute[test-nagios-config]', :delayed
end

By default, notifications are delayed until the very end of a Chef Infra Client run. You can run the action immediately with :immediately instead of :delayed

Templates

A cookbook template is an Embedded Ruby (ERB) template that is used to dynamically generate static text files. Templates can have Ruby expressions and statements and are a great way to manage configuration files. Use a template resource to add cookbook templates to recipes. The idea is to transfer files from the COOKBOOK_NAME/templates dir to a specific path located on a host that is running Chef Infra Client.

Syntax

template '/etc/motd' do
  source 'motd.erb'
  owner 'root'
  group 'root'
  mode '0755'
end

This means that /etc/motd is the location where the file gets created and motd.erb is the template used.

Knife

Knife Data Bag

Data bags store global variables as JSON data. Data bags can be loaded by a cookbook or accessed during a search.

Berkshelf

Berkshelf is a dependency manager for Chef cookbooks. You can depend on community cookbooks and have them safely included in your workflow. Berkshelf is included in ChefDK.

Setup

You can chef generate cookbook -b or --berks to create a Berksfile in the root of the cookbook. The Berksfile will be placed alongside the cookbook’s metadata.rb file (which has your cookbook’s dependencies to the metadata.rb file).

Metadata.rb

name 'my_first_cookbook'
version '0.1.0'
depends 'apt', ~> 5.0

Berksfile

source 'https://supermarket.chef.io'
metadata

If you run berks install, the apt cookbook will be downloaded from Supermarket into the cache. You can upload all cookbooks to your Chef Infra Server with berks upload

Berksfile

A Berksfile describes the set of sources and dependencies to use in a cookbook and is usually used with the berks command. A Berksfile is a Ruby file that has sources, dependencies, and options specified.

source "https://supermarket.chef.io"
metadata
cookbook "NAME" [, "VERSION_CONSTRAINT"] [, SOURCE_OPTIONS]

Source Keyword

A source defines where Berkshelf should look for cookbooks. Sources are processed in the order that they are defined in and stops processing as soon as a suitable cookbook is found. Sources can be Supermarket, Chef Infra Server, or local Chef repository.

To add a private Supermarket, which will be preferred:
source "https://supermarket.example.com"
source "https://supermarket.chef.io"
To add a Chef Infra Server:

source "https://supermarket.chef.io"
source :chef_server
To add a local Chef repository:

source "https://supermarket.chef.io"
source chef_repo: ".."

The location and authentication details for the Chef Infra Server is taken from the user’s config.rb by default

William Liu

github.com/williamqliu