High Availability: AWS (DEPRECATED)

Warning

This topic is deprecated as of the 12.9 release of the Chef servver. For the latest information on high availability and how to set up a highly-available server cluster, see High Availability: Backend Cluster .

This topic describes how to set up the Chef server for high availability in Amazon Web Services (AWS).

Note

This feature is included as part of the Chef Automate license agreement and is available via subscription.

Prerequisites

Before installing the Chef server software, perform the following steps:

  1. Use an Amazon Virtual Private Cloud (VPC). Amazon EC2-Classic is not supported.
  2. Create appropriate security groups to contain the backend instances. The only requirement for the Chef server is that ICMP is permitted between the two backend instances; Keepalived requires it for communication and heartbeat.
  3. Launch two servers, one for the primary backend Chef server and the other for the secondary backend Chef server. Use the same Amazon Machine Images (AMI) so that both backend servers have identical platform and versions. The servers must be in the same availability zones.
  4. Create an Amazon Elastic Block Store (EBS) volume to store the Chef server’s data. It is recommended that you use an EBS Provisioned IOPS volume type, with the maximum IOPS ratio for the size of volume.
  5. Choose an IP address for the backend virtual IP (VIP). It must reside in the same network segment as the backend instances and must be assignable as a secondary private IP address to the primary backend Chef server. The IP address is specified in the chef-server.rb file. During installation, the high-availability plugin will automatically assign the VIP to the elastic network interface (ENI) for the primary instance.
  6. Create an Identity and Access Management (IAM) user with at least the permissions documented in the reference section. Record this user’s access and secret keys; these will be used in the chef-server.rb configuration file.
  7. Enable sticky sessions on the load balancer. Configure all instances of the Chef management console to point to one Redis load balancer instance.

Primary Backend

Use the following steps to set up the primary backend Chef server:

  1. Create an Amazon Elastic Block Store (EBS) volume and attach it to the primary backend.

  2. Download the packages from http://downloads.chef.io/chef-server/ and http://downloads.chef.io/chef-ha/.

  3. Install the chef-server-core package. For Red Hat and CentOS 6:

    $ rpm -Uvh /tmp/chef-server-core-<version>.rpm

    For Ubuntu:

    $ dpkg -i /tmp/chef-server-core-<version>.deb

    After a few minutes, the Chef server will be installed.

  4. Install chef-ha package. For Red Hat and CentOS 6:

    $ rpm -Uvh /tmp/chef-ha-<version>.rpm

    For Ubuntu:

    $ dpkg -i /tmp/chef-ha-<version>.deb

  5. Create a file named chef-server.rb that is located in the /etc/opscode/ directory. See the chef-server.rb section below for an example of the settings and values that are required. The ha['ebs_device'] setting must specify the actual /dev device name that is reported by the machine’s kernel, which may not be the same value that is reported by Amazon Web Services (AWS). For example, Amazon Web Services (AWS) may refer to a volume as /dev/sdf through the management console, but to the Linux kernel on the instance, it may appear as /dev/xvdf.

  6. Install logical volume manager (LVM) tools. For Red Hat and CentOS 6:

    $ sudo yum install lvm2

    For Ubuntu:

    $ sudo apt-get install lvm2

  7. Create a physical volume, volume group, and logical volume with the following series of commands. The volume group and logical volume names must be chef and data, respectively.

    $ sudo pvcreate /dev/xvdf

    then:

    $ sudo vgcreate chef /dev/xvdf

    then:

    $ sudo lvcreate -l 85%VG -n data chef

  8. Format and mount the new volume with the following series of commands:

    $ sudo mkdir -p /var/opt/opscode/drbd/data

    then:

    $ sudo mkfs.ext4 /dev/mapper/chef-data

    and then:

    $ sudo mount /dev/mapper/chef-data /var/opt/opscode/drbd/data
  9. Reconfigure the Chef server and the Chef management console (standalone and frontend group members

    of a High Availabilty installation):

    $ sudo chef-server-ctl reconfigure
$ sudo chef-manage-ctl reconfigure

    This will reconfigure the Chef server, start Keepalived, assign the VIP IP address as a secondary address on the elastic network interface (ENI), and then configure the machine as the primary backend server.

  10. Verify the machine is the primary backend server:

    $ sudo chef-server-ctl ha-status

    This should display a screen of output indicating that the server is PRIMARY and that all services are running.

    Additionally, you may run the following command to verify that the VIP IP address is configured on the Ethernet interface:

    $ ip addr list dev eth0

    Warning

    Do not use the ifconfig command as it will not show all aliases.

chef-server.rb

Each Chef server in a high availabilty configuration must have an identical chef-server.rb file that is located in the /etc/opscode/ directory on each server. This file describes the topology of the high availability configuration. On the primary backend server, create a file named chef-server.rb and save it in the /etc/opscode/ directory.

Add the following settings to the chef-server.rb file:

  1. Define the topology type:

    topology "ha"

  2. Define the primary backend server:

    server "FQDN",
  :ipaddress => "IP_ADDRESS",
  :role => "backend",
  :bootstrap => true,
  :cluster_ipaddress => "CLUSTER_IPADDRESS"

    Replace FQDN with the FQDN of the server and IP_ADDRESS with the IP address of the server. The role is a backend server is "backend". If the backend server is used to bootstrap the Chef server installation, replace CLUSTER_IPADDRESS with the IP address of the interface that is used for cluster communications. For example, the same IP address that is used by Keepalived and DRBD. If the Chef server is not used to bootstrap the Chef server installation, exclude the :cluster_ipaddress entry.

  3. Define the secondary backend server:

    server "FQDN",
  :ipaddress => "IPADDRESS",
  :role => "backend",
  :cluster_ipaddress => "CLUSTER_IPADDRESS"

    Replace FQDN with the FQDN of the server, and IPADDRESS with the IP address of the server. Replace CLUSTER_IPADDRESS with the IP address of the server’s interface assigned for cluster communications. If no such interface is configured, exclude the cluster_ipaddress entry.

  4. Define the backend virtual IP address:

    backend_vip "FQDN",
  :ipaddress => "IP_ADDRESS",
  :device => "eth0",

    Replace FQDN with the FQDN of the server. Replace IP_ADDRESS with the virtual IP address of the server. The :device parameter should be the ethernet interface to which the floater virtual IP address will bind. This is typically the public interface of the server.

  5. Define each frontend server:

    server "FQDN",
  :ipaddress => "IP_ADDRESS",
  :role => "frontend"

    Replace FQDN with the FQDN of the frontend server. Replace IP_ADDRESS with the IP address of the frontend server. Set :role to "frontend".

    Add separate entry in the chef-server.rb file for each frontend server.

  6. Define the API FQDN:

    api_fqdn "FQDN"

    Replace FQDN with the FQDN of the load balanced virtual IP address, which should be equal to the FQDN for the service URI that is used by the Chef server.

  7. Reconfigure the Chef server and the Chef management console (standalone and frontend group members

    of a High Availabilty installation):

    $ sudo chef-server-ctl reconfigure
$ sudo chef-manage-ctl reconfigure

Secondary Backend

Use the following steps to set up the secondary backend Chef server:

  1. Install the chef-server-core package. For Red Hat and CentOS 6:

    $ rpm -Uvh /tmp/chef-server-core-<version>.rpm

    For Ubuntu:

    $ dpkg -i /tmp/chef-server-core-<version>.deb

    After a few minutes, the Chef server will be installed.

  2. Install chef-ha package. For Red Hat and CentOS 6:

    $ rpm -Uvh /tmp/chef-ha-<version>.rpm

    For Ubuntu:

    $ dpkg -i /tmp/chef-ha-<version>.deb

  3. Install logical volume manager (LVM) tools. For Red Hat and CentOS 6:

    $ sudo yum install lvm2

    For Ubuntu:

    $ sudo apt-get install lvm2

  4. Create the /etc/opscode/ directory, and then copy the contents of the entire /etc/opscode directory from the primary server, including all certificates and the chef-server.rb.

  5. Reconfigure the Chef server and the Chef management console (standalone and frontend group members

    of a High Availabilty installation):

    $ sudo chef-server-ctl reconfigure
$ sudo chef-manage-ctl reconfigure

    This will reconfigure the Chef server, start Keepalived, and configure it as the secondary backend server.

  6. Verify the secondary backend server:

    $ sudo chef-server-ctl ha-status

    This should indicate that the server is BACKUP.

Verify Failover

To verify that failover is working, stop Keepalived on the primary server.

  1. To watch the failover occur as it happens, run the following command in terminal windows on both the primary and secondary backend servers prior to stopping Keepalived:

    $ watch -n1 sudo chef-server-ctl ha-status

    in terminal windows on both the primary and secondary servers prior to stopping Keepalived.

  2. Stop Keepalived on the primary backend server:

    $ sudo chef-server-ctl stop keepalived

    A cluster failover should occur.

  3. After a successful failover, restart Keepalived on the primary backend server:

    $ sudo chef-server-ctl start keepalived

    The primary has now become the secondary, and vice-versa. If you wish to fail back to the original primary, repeat these using the new primary.

Frontend Installation

Use the following steps to set up each frontend Chef server:

  1. Install the chef-server-core package. For Red Hat and CentOS 6:

    $ rpm -Uvh /tmp/chef-server-core-<version>.rpm

    For Ubuntu:

    $ dpkg -i /tmp/chef-server-core-<version>.deb

    After a few minutes, the Chef server will be installed. The Chef high availability package is not required on front end machines.

  2. Create the /etc/opscode/ directory, and then copy the entire contents of the /etc/opscode directory from the primary backend server, including all certificates and the chef-server.rb file.

  3. Reconfigure the Chef server and the Chef management console (standalone and frontend group members

    of a High Availabilty installation):

    $ sudo chef-server-ctl reconfigure
$ sudo chef-manage-ctl reconfigure

  4. Run the following command:

    $ sudo chef-server-ctl start

  5. Run the following command to create an administrator:

    $ chef-server-ctl user-create USER_NAME FIRST_NAME LAST_NAME EMAIL 'PASSWORD' --filename FILE_NAME

    An RSA private key is generated automatically. This is the user’s private key and should be saved to a safe location. The --filename option will save the RSA private key to a specified path.

    For example:

    $ chef-server-ctl user-create stevedanno Steve Danno [email protected] 'abc123' --filename /path/to/stevedanno.pem

  6. Run the following command to create an organization:

    $ chef-server-ctl org-create short_name 'full_organization_name' --association_user user_name --filename ORGANIZATION-validator.pem

    The name must begin with a lower-case letter or digit, may only contain lower-case letters, digits, hyphens, and underscores, and must be between 1 and 255 characters. For example: 4thcoffee.

    The full name must begin with a non-white space character and must be between 1 and 1023 characters. For example: 'Fourth Coffee, Inc.'.

    The --association_user option will associate the user_name with the admins security group on the Chef server.

    An RSA private key is generated automatically. This is the chef-validator key and should be saved to a safe location. The --filename option will save the RSA private key to a specified path.

    For example:

    $ chef-server-ctl org-create 4thcoffee 'Fourth Coffee, Inc.' --association_user stevedanno --filename /path/to/4thcoffee-validator.pem
  7. Reconfigure the Chef server and the Chef management console (standalone and frontend group members

    of a High Availabilty installation):

    $ sudo chef-server-ctl reconfigure
$ sudo chef-manage-ctl reconfigure

Enable Features

Enable additional features of the Chef server! The packages may be downloaded directly as part of the installation process or they may be first downloaded to a local directory, and then installed.

Use Downloads

The install subcommand downloads packages from https://packages.chef.io/ by default. For systems that are not behind a firewall (and have connectivity to https://packages.chef.io/), the Chef management console package can be installed as described below:

Chef Manage

Use Chef management console to manage data bags, attributes, run-lists, roles, environments, and cookbooks from a web user interface.

On each front end server in the Chef server configuration, run:

$ chef-server-ctl install chef-manage

then:

$ chef-server-ctl reconfigure

and then:

$ chef-manage-ctl reconfigure

This updates the Chef server and creates the /etc/opscode-manage/secrets.rb file. When running the Chef management console 1.11 (or higher), copy the secrets.rb file in the /etc/opscode-manage directory on one of the frontend servers to the same directory on each of the other frontend servers, and then rerun chef-manage-ctl reconfigure so the copied /etc/opscode-manage/secrets.rb file gets used correctly.

Note

Starting with the Chef management console 2.3.0, the Chef MLSA must be accepted when reconfiguring the product. If the Chef MLSA has not already been accepted, the reconfigure process will prompt for a yes to accept it. Or run chef-manage-ctl reconfigure --accept-license to automatically accept the license.

Use Local Packages

The install subcommand downloads packages from https://packages.chef.io/ by default. For systems that are behind a firewall (and may not have connectivity to packages.chef.io), these packages can be downloaded from https://downloads.chef.io/chef-manage/, and then installed manually. First download the package that is appropriate for the platform, save it to a local path, and then run the install command using the --path option to specify the directory in which the package is located:

$ chef-server-ctl install PACKAGE_NAME --path /path/to/package/directory

For example:

$ chef-server-ctl install chef-manage --path /root/packages

The chef-server-ctl command will install the first chef-manage package found in the /root/packages directory.

Install Reporting

To set up the Reporting server:

  1. Install the package on each frontend and backend Chef server:

    $ chef-server-ctl install opscode-reporting

  2. Reconfigure the Chef server on the backend primary server (bootstrap):

    $ chef-server-ctl reconfigure

  3. Reconfigure the Reporting server on the backend primary server (bootstrap):

    $ opscode-reporting-ctl reconfigure

    Note

    Starting with Reporting 1.6.0, the Chef MLSA must be accepted when reconfiguring the product. If the Chef MLSA has not already been accepted, the reconfigure process will prompt for a yes to accept it. Or run opscode-reporting-ctl reconfigure --accept-license to automatically accept the license.

  4. Copy the entire /etc/opscode-reporting directory from the backend primary server to all frontend and backend servers. For example, from each server run:

    $ scp -r <Bootstrap server IP>:/etc/opscode-reporting /etc

    or from the backend primary server:

    $ scp -r /etc/opscode-reporting <each servers IP>:/etc

  5. Reconfigure any Chef server on which Reporting services have been installed:

    $ chef-server-ctl reconfigure

  6. Reconfigure Reporting services on each server:

    $ opscode-reporting-ctl reconfigure

  7. Verify the installation:

    $ opscode-reporting-ctl test

Install Push Jobs

To set up the Chef push jobs server for a high availability configuration:

  1. Install the package on all servers that are running the Chef server. For example on Ubuntu:

    $ sudo dpkg -i opscode-push-jobs-server_2.1.0-1_amd64.deb

  2. Reconfigure the primary backend Chef push jobs server:

    $ opscode-push-jobs-server-ctl reconfigure

  3. Copy the entire /etc/opscode-push-jobs-server directory from the backend primary to all frontend and backend servers. For example, from each server run:

    $ scp -r <Bootstrap server IP>:/etc/opscode-push-jobs-server /etc

    or from the backend primary server:

    $ scp -r /etc/opscode-push-jobs-server <each servers IP>:/etc

  4. TCP protocol ports 10000 and 10003 must be open. These are the heartbeat and command ports respectively. They allow the Chef push jobs server to communicate with the Chef push jobs clients. In a configuration with both frontend and backend servers, these ports only need to be open on the backend servers. The Chef push jobs server waits for connections from the Chef push jobs client (and never makes a connection to a Chef push jobs client).

  5. Reconfigure the remaining Chef push jobs servers:

    $ opscode-push-jobs-server-ctl reconfigure

  6. Run the following command on each of the backend servers:

    $ chef-server-ctl reconfigure

    This ensures that the Keepalived scripts are regenerated so they are aware of Chef push jobs.

  7. Restart all servers on which Chef push jobs will run:

    $ chef-server-ctl restart opscode-pushy-server

  8. Verify the installation:

    $ opscode-push-jobs-server-ctl test

Reference

The following sections show the Chef high availability settings as they appear in a chef-server.rb file and required permissions of the user in Identity and Access Management (IAM).

chef-server.rb

The following example shows a chef-server.rb file:

topology "ha"
ha['provider'] = 'aws'
ha['aws_access_key_id'] = '[DELETED]'
ha['aws_secret_access_key'] = '[DELETED]'
ha['ebs_volume_id'] = 'vol-xxxxx'
ha['ebs_device'] = '/dev/xvdf'

server 'ip-172-31-24-97.us-west-1.compute.internal',
  :ipaddress => '172.31.24.97',
  :role => 'backend',
  :bootstrap => true

server 'ip-172-31-24-98.us-west-1.compute.internal',
  :ipaddress => '172.31.24.98',
  :role => 'backend'

backend_vip 'ip-172-31-24-180.us-west-1.compute.internal',
  :ipaddress => '172.31.24.180',
  :device => 'eth0',
  :heartbeat_device => 'eth0'

server 'ip-172-31-30-47.us-west-1.compute.internal',
  :ipaddress => '172.31.30.47',
  :role => 'frontend'

api_fqdn 'ec2-54-183-175-188.us-west-1.compute.amazonaws.com'

Identity and Access Management (IAM)

The following example shows Identity and Access Management (IAM) access management settings that are required for Chef high availability:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeVolumes",
        "ec2:AttachVolume",
        "ec2:DetachVolume",
        "ec2:AssignPrivateIpAddresses"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

It is possible to further restrict access using a more sophisticated policy document. For example, administrators may choose to permit the Identity and Access Management (IAM) user only to attach/detach the volume ID associated with the Chef server data volume, and not all volumes.

© Chef Software, Inc.
Licensed under the Creative Commons Attribution 3.0 Unported License.
The Chef™ Mark and Chef Logo are either registered trademarks/service marks or trademarks/servicemarks of Chef, in the United States and other countries and are used with Chef Inc's permission.
We are not affiliated with, endorsed or sponsored by Chef Inc.
https://docs-archive.chef.io/release/server_12-8/install_server_ha_aws.html