Delivery Truck

[edit on GitHub]

Workflow is a legacy feature for Chef Automate, which was designed for managing changes to both infrastructure and application code, giving your operations and development teams a common platform for developing, testing, and deploying cookbooks, applications, and more.

delivery-truck is a cookbook for Chef Automate that should be a dependency of every recipe in a build-cookbook, which is effectively a project-specific wrapper cookbook for the delivery-truck cookbook. The delivery-truck cookbook defines a set of recipes that correspond to the phases and stages in the Chef Automate pipeline and help ensure good default build-cookbook behavior. Chef recommends including the delivery-truck cookbook in all recipes in a build-cookbook.

‘delivery-truck’ Recipes

The following recipes are available by default in the delivery-truck cookbook:

default.rb

Use the default.rb recipe to configure a project on a build node. This recipe is run by the Chef Infra Client as the root user and is a standard default recipe. The Chef Infra Client may use this recipe to configure this project on any node, whether or not it’s part of a Chef Automate pipeline.

deploy.rb

Use the deploy.rb recipe to define how artifacts are published to one (or more) nodes after they are built successfully. The contents of this recipe are project-specific.

functional.rb

Use the functional.rb recipe to run a set functional tests that are specific to this project. The tests are run on a single build node and should target and/or trigger tests against the set of nodes that are updated when this artifact deploys.

lint.rb

Use the lint.rb recipe to run linting and other static analysis tools against a project’s source code.

provision.rb

Use the provision.rb recipe to build any infrastructure that is necessary to run an application. This recipe will discover all metadata.rb and/or metadata.json files that are located in the project’s root directory, plus any cookbook directories located under cookbooks/<project_cookbooks>.

publish.rb

Use the publish.rb recipe to make any artifact generated by this project available to other phases in the Chef Automate pipeline.

quality.rb

Use the quality.rb recipe to run additional code quality and reporting tools.

security.rb

Use the security.rb recipe to execute security tests against a project’s source code.

smoke.rb

Use the smoke.rb recipe to run smoke tests against deployed build artifacts to ensure they were deployed correctly and are minimally functional.

syntax.rb

Use the syntax.rb recipe to verify that changes result in syntactically correct code. This process may involve compiling the code or running a validator for interpreted languages.

unit.rb

Use the unit.rb recipe to run unit tests for the project.

Create a build-cookbook

Pull the delivery-truck and delivery-sugar cookbooks into a build-cookbook. This requires editing the Berksfile, and then updating the metadata.rb file.

Edit the Berksfile

The Berksfile for a build-cookbook is located at .delivery/build-cookbook/Berksfile. Update it to include:

source "https://supermarket.chef.io"

metadata

cookbook 'delivery-truck', github: 'chef-cookbooks/delivery-truck'
cookbook 'delivery-sugar', github: 'chef-cookbooks/delivery-sugar'

This will ensures that the latest versions of the delivery-truck and delivery-sugar cookbooks are pulled into the build-cookbook every time a change is sent to the Chef Automate project pipeline.

Edit metadata.rb

The metadata.rb for a build-cookbook is located at .delivery/build-cookbook/metadata.rb. Update it to include:

depends 'delivery-truck'

This will ensure that the build-cookbook has a dependency on the delivery-truck cookbook.

Add delivery-truck to Recipes

A build-cookbook should define the same phases as the recipes included in the delivery-truck cookbook: default.rb, deploy.rb, functional.rb, lint.rb, provision.rb, publish.rb, quality.rb, security.rb, smoke.rb, syntax.rb, and unit.rb. For example, a build cookbook’s recipe directory should contain an identical list of recipes. For example, run:

$ ls .delivery/build-cookbook/recipes/

the list of recipes should be:

default.rb
deploy.rb
functional.rb
lint.rb
provision.rb
publish.rb
quality.rb
security.rb
smoke.rb
syntax.rb
unit.rb

Each recipe corresponds to a specific phase in the Chef Automate pipeline. The recipes in the build-cookbook should include the same-named recipe in the delivery-truck cookbook. For example, to include the lint.rb recipe from the delivery-truck cookbook, update the lint.rb recipe in the build-cookbook to add the following:

include_recipe 'delivery-truck::lint'

and then add to the unit.rb recipe:

include_recipe 'delivery-truck::unit'

…and so on for all of the recipes. This ensures that all of the default behavior for all of the phases for the entire pipeline is available to this build-cookbook.

Project Cookbooks

A project cookbook is a cookbook that is located within a project and is used to deploy that project’s software onto one (or more) nodes in the Chef Automate pipeline. These cookbooks are located in the /cookbooks directory, which should exist at the root of the project (similar to the .delivery directory).

The default.rb recipe in a project cookbook is executed by the Chef Infra Client on infrastructure nodes as the project moves through the Chef Automate pipeline. The provision.rb recipe discovers all metadata.rb and/or metadata.json files in the project, including those under the /cookbooks directory.

Single Cookbook

A project may use a single cookbook to tell the Chef Infra Client how to configure nodes in the Chef Automate pipeline.

Add Project Cookbook

Create a project cookbook. From the project’s root directory, do the following:

1. Create a branch:

   $ delivery checkout BRANCH_NAME
   

2. Generate a cookbook under /cookbooks in the project directory:

   $ chef generate cookbook PROJECT_NAME
   

   Review the metadata.rb file. It should be similar to:

   name 'my_project'
   maintainer 'The Authors'
   maintainer_email '[email protected]'
   license 'all_rights'
   description 'Installs/Configures my_project'
   long_description 'Installs/Configures my_project'
   version '0.1.0'
   

   where version '0.1.0' must be changed if files within the cookbook change. The version number is what gets promoted through the stages in the Chef Automate pipeline.

Configure default.rb

In the default.rb recipe, define how this project is to be deployed. This is a normal Chef recipe that is executed by the Chef Infra Client, so do the same in this recipe as you would do in any other.

Promote the Project

When a change to a project is submitted to Chef Automate, the provision.rb does the work of promoting the project to the various nodes in the Chef Automate pipeline.

To submit changes to Chef Automate, use commands similar to:

1. Update to match the working tree:

   $ git add -A
   

2. Commit the project:

   $ git commit -m "Let us deploy our app."
   

3. Review the changes in Chef Automate:

   $ delivery review
   

   This command will open the Chef Automate web UI, and then run unit, lint, and syntax tests. After the tests pass, the change may be approved. Once approved, the provision.rb recipe will deploy the project onto the acceptance stage’s infrastructure nodes.

4. After the change has built successfully through the Acceptance stage, approve the changes by selecting the Deliver button in the Chef Automate web UI. This sends the project to the Union, Rehearsal, and Delivered stages.

Update the Project

Update a file in the project, and then update the version number in the metadata.rb file. This ensures this cookbook is promoted, overwriting the old project cookbook, and then updating the project across each stage of the Chef Automate pipeline:

1. Check out the project from Chef Automate:

   $ delivery checkout master
   

2. Create a branch:

   $ delivery checkout BRANCH_NAME
   

3. Edit the version in the metadata.rb file:

   version '0.2.0'
   

   and then make the desired changes.

4. Update to match the working tree:

   $ git add -A
   

5. Add a commit message:

   $ git commit -m "Updated our project's code to version 0.2.0."
   

6. Review the changes in Chef Automate:

   $ delivery review
   

Multiple Cookbooks

Some projects need more than one project cookbook. Put as many cookbooks as necessary under the /cookbooks directory, which is located at the root of a project.

Each cookbook under the /cookbooks directory must have a valid cookbook structure. If the cookbook does not have a metadata.rb or metadata.json file, it will not be discovered by the provision.rb recipe. Consequently, that cookbook will not be used to configure nodes in the Chef Automate pipeline.

The default.rb recipes in all project cookbooks are executed by the Chef Infra Client on infrastructure nodes as the project moves through the Chef Automate pipeline. The default.rb recipe in the build-cookbook is run first, and then each default.rb recipe in each cookbook under /cookbooks is run (in alphabetical order, by cookbook name).

Project Applications

A project may be a binary, a package, or some other set of arbitrary information. The Chef Automate pipeline supports promoting projects through the pipeline using versioned attributes. This is known as a project application. A project application is a useful way to promote projects by using a set of attributes that are pinned to a specific version, and then using those same versioned attributes when deploying software to various stages in the Chef Automate pipeline.

Configure Project Application

Project applications are defined in the publish.rb recipe in a build-cookbook using the define_project_application helper method, and then in the deploy.rb recipe using the get_project_application method. The publish phase happens at the end of the build stage. It is at this point where the project application version is pinned, uploaded to the Chef Infra Server as a data bag item, and then used through the remaining stages.

The define_project_application helper method is available from the delivery-sugar cookbook, which is a dependency of the delivery-truck cookbook. This helper is available when the publish.rb recipe has include_recipe 'delivery-truck::publish' defined.

To define a project application, do the following:

1. Open the publish.rb recipe in the build-cookbook and edit it to contain:

   define_project_application(
     <app_name>,
     <app_version>,
     [ 'attribute',
       'attribute',
       ... ]
   )
   

   where

   • <app_name> is the name of the project application
   • <app_version> is version number to which the project application is pinned
   • 'attribute' is Hash of attributes associated with this version; each attribute is defined as a key-value pair: 'key = value'

2. Set up the build-cookbook to know about this application. If the .delivery/build-cookbook/attributes/default.rb directory and/or file does not exist, create it. Add the following to it:

   default['delivery']['project_apps'] = ['<app_name>', '<app_name>', ...]
   

   where <app_name> is a list of one (or more) applications this build-cookbook should be aware of.

3. Open the default.rb recipe in the build-cookbook and edit it to contain:

   { 'hash_of_attributes' } = get_project_application(<app_name>)
   

   where 'hash_of_attributes' is a list of one (or more) attributes defined in the define_project_application block.

   Do not pass 'id', 'version', or 'name' as part of the 'hash_of_attributes' as these are already defined in the define_project_application block, are pulled in automatically by the get_project_application helper method, and will overwrite any value specified in the hash.

Example Project Application

This example shows how to use project applications to deploy a package into a .deb file during the deploy phase. (This example assumes a Chef Automate project exists with a properly configured build-cookbook.)

1. Open the publish.rb recipe in the build-cookbook and edit to look like the following:

   include_recipe 'delivery-truck::publish'
   
   # Generate your artifact and document it's location on a download server.
   artifact_location = <generated_artifact_location>
   
   # It's recommended to generate a checksum from your package too.
   artifact_checksum = <package_checksum>
   
   # Version the artifact based on the current date.
   artifact_version = Time.now.strftime('%F_%H%M')
   
   # Name your application.
   name = "<app_name>"
   
   project_app_attributes = {
     'artifact_location' => artifact_location,
     'artifact_checksum' => artifact_checksum
   }
   
   define_project_application(
     name,
     artifact_version,
     project_app_attributes
   )
   

2. In the publish.rb recipe, update <generated_artifact_location> and <package_checksum> to be correct for this project.

3. Set up the build-cookbook to know about this application. Add the following to .delivery/build-cookbook/attributes/default.rb:

   default['delivery']['project_apps'] = ["<app_name>"]
   

   where <app_name> is the same value as the name of the application in the publish.rb file. When the publish phase is run, an application is created, versioned by timestamp, and including all of the information needed to install that version of the application. The provisioning code in delivery-truck will automatically pin based on this version.

4. Configure the build-cookbook to know how to install the application. Add the following to .delivery/build-cookbook/deploy.rb:

   app_attributes = get_project_application("<APPLICATION_NAME>")
   
   # Download your package.
   remote_file "/tmp/latest_package.deb" do
     source   app_attributes['artifact_location']
     checksum app_attributes['artifact_checksum']
     action :create
   end
   
   # Install it onto your build infrastructure.
   package app_attributes['name'] do
     source "/tmp/latest_package.deb"
     action :install
   end
   

Validate the Installation

The surest way to validate a Chef Automate installation is to create a cookbook, and then submit it to Chef Automate to kick off a new build in the pipeline.

If a project is a cookbook, we recommend starting with delivery-truck, an open source build cookbook created for driving cookbook pipelines in Chef Automate. You can customize some aspects of delivery-truck through your project’s .delivery/config.json. To have more control or to opt-out of some of the behavior of delivery-truck, create a wrapper build cookbook.

This topic describes the recommended setup for a Chef cookbook project using Chef Automate.

The following example shows how to create a cookbook, with project and pipeline, configure it to be built with Chef Automate, and then imported it into Chef Automate itself. From your workstation as user with admin privileges on the Chef Automate server, do the following:

1. Make a working directory (workspace in the example):

   $ mkdir ~/workspace && cd ~/workspace
   

2. Setup the Delivery CLI to, by default, contact the Chef Automate server at SERVER, with a default ENTERPRISE and ORGANIZATION:

   $ delivery setup --server=SERVER --ent=ENTERPRISE --org=ORGANIZATION --user=USERNAME
   

   Note

   The server, enterprise, organization, and user must already exist.
   

3. Create a cookbook, then navigate into it:

   $ chef generate cookbook NEW-COOKBOOK-NAME
   $ cd NEW-COOKBOOK-NAME
   

   This uses the Chef development kit to generate a new cookbook, including a default recipe and default ChefSpec tests.

4. Create an initial commit (use git status to verify the change) on the “master” branch:

   $ git add .
   

   $ git commit -m 'Initial Commit'
   

   Running chef generate initialized a git repository automatically for this cookbook. If you created the build cookbook manually, initialize the git repository with the git init command.

5. Initialize the cookbook for Chef Automate:

   $ delivery init
   

   This creates a new project in Chef Automate, pushes the master branch, creates a feature branch, generates a default Chef Automate project configuration file, pushes the first change for review, and then opens a browser window that shows the change.

6. Now that you have initialized your project, it is recommended that you integrate the delivery-truck cookbook with your project. Delivery Truck can ensure good build cookbook behavior as well as provide you with recipes already set up to test your project cookbooks and applications.

Using delivery-truck in air-gapped environment

Chef Automate can be set up to deploy cookbooks and applications in an air-gapped environment and this section describes how to set up a basic cookbook to be delivered through Chef Automate using the delivery-truck cookbook in that environment.

By default, the delivery-truck cookbook is configured for use with Chef Automate-backed cookbook projects.

Prerequisites

• Ensure you have a private Supermarket installed, setup, and running. See Install Private Supermarket for more information.
• Ensure you have a Chef Infra Server with the Chef Identity authentication/authorization service configured, a Chef Automate server setup that references your private Supermarket, and at least one Chef Automate build node/runner installed, setup, and running. See Install Chef Automate and Chef Identity for more information.
• Ensure you have created a project in Chef Automate. Follow these instructions to Set Up Projects.
• Ensure you have ChefDK installed on your workstation.

Share cookbooks with your private Supermarket

To use delivery-truck and its dependency, delivery-sugar, you must first share them with a private Supermarket that is authenticated with your Chef Infra Server.

1. From a workstation, create a cookbooks directory, $COOKBOOKS_DIR:

   mkdir -p $COOKBOOKS_DIR
   

2. Clone the delivery-truck cookbook and its dependency deliver-sugar from GitHub:

   cd $COOKBOOKS_DIR
   git clone https://github.com/chef-cookbooks/delivery-sugar.git
   git clone https://github.com/chef-cookbooks/delivery-truck.git
   

3. To ensure your private Supermarket does not try to connect to third-party services, log into it and set the AIR_GAPPED environment variable to 'true' in the /etc/supermarket/supermarket.rb file.

   default['supermarket']['air_gapped'] = 'true'
   

4. Save your changes and close the file.

5. Reconfigure your private Supermarket.

   supermarket-ctl reconfigure
   

6. Share the delivery-truck and delivery-sugar cookbooks with your private Supermarket using the knife command-line tool. If you have not configured knife to share cookbooks with your private Supermarket, see Upload a Cookbook before running the following knife subcommands.

   knife supermarket share 'delivery-truck'
   knife supermarket share 'delivery-sugar'
   

Generate a cookbook

1. Use Chef Workstation’s cookbook generator command to create a default cookbook directory structure called my_cookbook.

   chef generate cookbook my_cookbook
   

2. Run delivery init in your my_cookbook local directory to create a new project in Chef Automate and push your first change for review.

   cd my_cookbook
   delivery init
   

3. Finally, check out the added files and commit your changes.

Use the delivery-truck cookbook in your project

From the root of your project’s directory, do the following:

1. Modify the build cookbook’s Berksfile to reference delivery-truck and delivery-sugar. By default, this file is located at .delivery/build-cookbook/Berksfile.

   source 'https://your_private_supermarket_url'
   
   metadata
   
   group :delivery do
     cookbook 'delivery-sugar'
     cookbook 'delivery-truck'
   end
   

2. Modify the build cookbook’s metadata to include delivery-truck. By default, this file is located at .delivery/build-cookbook/metadata.rb.

   depends 'delivery-truck'
   

3. Edit your build cookbook’s recipes to include the corresponding delivery-truck recipe.

   # Cookbook Name:: $BUILD_COOKBOOK_NAME
   # Recipe:: $RECIPE
   #
   # Copyright (c) 2016 The Authors, All Rights Reserved.
   
   include_recipe "delivery-truck::$RECIPE"
   

   By default, each build cookbook recipe $RECIPE is located at .delivery/build-cookbook/recipes/$RECIPE.rb.

4. Increment your build cookbook’s version in the cookbook’s metadata file.

5. Commit your changes and run delivery review. Changes to your cookbook project can now be managed by your Chef Automate cluster.