Adopt Devops & Cloud

Chef - Returning EC2 Metadata for Search

How do you automatically clean up a Chef server to remove nodes that have been decommissioned? This simple recipe helps you do so at scale.

by Dave Tashner

It’s pretty common for enterprises to have multiple provisioning platforms at their disposal, whether that’s a private cloud on VMWare, a public cloud in AWS, Azure, or Google Compute Engine, or bare metal boxes in a datacenter. This complexity unavoidably creates issues when running Chef at scale; for instance, how do you automatically clean up public cloud nodes from the Chef server which may be waking up to serve a purpose for maybe an hour (or less) before their end of life is reached? 

A quick Google search will return a solution like this one, where your favorite AWS SDK or CLI is used to query various AWS VPCs and the Chef server via AWS instance-id, compare the results, and delete any nodes from the Chef server which are not running in AWS. This works well because instance-ids in AWS are globally unique. The only slight problem with this solution is that it is predicated on instance-id being available for search on the Chef server, and that particular attribute is not a default Ohai attribute (Ohai is the system profiler RubyGem in the Chef world that returns ‘automatic’ attributes about your node such as IP address, hostname, FQDN, etc). Ohai does not automatically return EC2 metadata (including instance-id), because Chef is not an AWS-specific configuration management tool. Because Ohai does not return this information by default, metadata information such as EC2 instance-id is not available for search because the information is not available in the node object and is therefore not indexed for search on the Chef server.

However, Chef does provide an EC2 Ohai plugin by default that will pull this information for you (default location on my test CentOS 7 box is

/opt/chef/embedded/apps/ohai/lib/ohai/plugins/ec2.rb

Essentially, including this plugin will force Ohai to make a call to the EC2 metadata service (169.254.169.254) and return the instance metadata (including instance-id) as part of the node object, which is then saved to the Chef server and indexed for search by Apache Solr. A simple recipe like the one below can run on Windows, RedHat, and Ubuntu to return the EC2 metadata for AWS instances, and it has no impact on non-AWS nodes (those nodes just return blank EC2 information).

case node['platform_family']
  when 'rhel', 'debian'
    #Set *nix-specific directory
    hints = "/etc/chef/ohai/hints"

    file "create_hint_file" do
      path "#{hints}/ec2.json"
      action :nothing
      notifies :reload, "ohai[get_instance_metadata]"
    end

  when 'windows'
    #Set Windows-specific directory
    hints = "C:\\chef\\ohai\\hints"

    file "create_hint_file" do
      path "#{hints}\\ec2.json"
      action :nothing
      notifies :reload, "ohai[get_instance_metadata]"
    end
end

#Create the hints directory on Windows and Linux and immediately create the hints file
#before Ohai is reloaded
directory hints do
  action :create
  recursive true
  notifies :create, "file[create_hint_file]", :immediately
end

#Don't refresh unless notified
ohai 'get_instance_metadata' do
  action :nothing
  plugin 'ec2'
end

Since this is a recipe instead of a shell command, it’s easy to include the recipe in your organization’s base role so that it is applied to all nodes. At the next check in, EC2 metadata will be available for search for all nodes in the organization, making it easy to run the previously referenced cleanup scripts.

A word of caution here; because this EC2 metadata is somewhat verbose, the impact on the size of your node objects can be significant. Be aware of this additional node object data and make sure to monitor your Chef server in case it starts to become unhappy with the additional data, especially if you have a large number of nodes under management. It may be necessary to rewrite a custom Ohai EC2 plugin that only returns instance-id instead of the entire EC2 metadata.

Learn more about our DevOps and Cloud solutions.

Dave Tashner
Dave Tashner
Senior Consultant
Contact Dave