Adopt Cloud and DevOps

Secrets Management With AWS

Cloud Security Architect Don Mills gives a tutorial on secrets and credentials management using AWS services.

by Don Mills

One of the more in-depth challenges we see as we help clients develop Continuous Integration and Continuous Deployment pipelines has do to with secrets and credentials management. Often CI/CD pipelines will have infrastructure deployment pieces that require things like database passwords, user logins, or other information that you might not want to be widely accessible, even within a single organization.

While there are existing solutions in the ecosystem that are popular (Vault by Hachicorp being a prime example), most of these require some server infrastructure to be set up to handle the backend. This can add not only additional cost and complexity, but also operational overhead - now you have to run yet another server, and if you are using it as a core piece of your infrastructure then it had better be redundant and highly-available. You will also (particularly with Vault) need to install client pieces on hosts that need to access the secrets...

Let me give you an example of the problem first. Suppose you are building a Development environment dynamically in AWS as part of your CI/CD pipeline. As a component of this you are using Chef or Puppet (or your favorite configuration management tool) to configure the servers. You have to feed in database passwords as part of the configuration. If you hard-code them in your configuration scripts, you will expose them to anyone that can read that script (fellow GitHub users perhaps, or people with access to the configuration management server, etc). If you were using CloudFormation and hard-coded them in that, you could expose them to anyone on your AWS account unless you tightly control CloudFormation access in IAM, and even then anyone with the right IAM permissions (perhaps you have multiple teams on the same account using CloudFormation) could see the stack information.

You can encrypt the secrets, but then there is the issue of handling the key necessary to decrypt them on the end host. If you encrypt something to secure it but then have the key lying around in Chef recipes, or in a CloudFormation UserData section, then all you have done is what I like to call "kicking the can down the road a bit". Anyone who gets the still available key can just decrypt the original secret data. So what we need is a method to encrypt the secrets, and then secure the key so that it can be freely transferred around, but only authorized entities can use it...

A possible solution

So first, let's not kid ourselves in that there is a perfect solution for this problem. If you encrypt something, you have to have a key to decrypt it. This is true whether you are doing symmetrical encryption (where the same key encrypts and decrypts) or asymmetrical (where one key encrypts and another decrypts). At some point, you have to have a key usable to the entity that needs the secret data. But what we can do is take steps to protect that key from unauthorized usage and yet make it simple enough that anyone can implement our protection in a repeatable manner without too much overhead.

After much discussion with my team members on this problem I spent a lot of time thinking about a solution. And using AWS services (as that was the environment we were working in at the time) I developed the strategy and code I am going to present here. I should also state definitively right here and now that I am not a programmer by trade, and certainly not a Ruby programmer by any means (in fact this is some of the first Ruby code I have ever written)...so you will have to pardon any programmatic shortcomings if you are such an expert. But I can say that it works, and it works at a large scale in production environments.

The building blocks

The basic elements of this solution are two AWS services, one of which almost everyone has been exposed to - the Simple Storage Service or S3 - and one which people have heard of but is not as widely used - the Key Management Service or KMS.

S3, of course, is Amazon's highly redundant storage service and the location where our secrets are placed. You can (and should) control access to S3 via IAM roles assigned to your EC2 instances, which can tighten down access to specific buckets and even a folder inside a bucket. We typically store the secrets in a bucket dedicated to the CI/CD pipeline for a specific application.

And then there is KMS. KMS is typically used by users to provide server-side encryption keys for other AWS services, such as S3 buckets or EBS volumes. But server-side encryption does not protect the data once the EBS volume is mounted, or the bucket is accessible via S3 permissions. It primarily exists to protect data you put in AWS services from unauthorized access from anyone with physical access to the storage. What we want is client-side encryption where the data is encrypted before it even gets to S3, and is decrypted after it is retrieved. And for that, the KMS APIs provide a series of interesting options. Let's look at the KMS API for the Ruby SDK.

First, there is an encrypt function. This sounds like a possible answer, but the important fact here is that this function has a limit of 4K worth of data. If your secrets or important data could ever be over 4K in size, this will not be a viable option.

But as we look farther we see generate_data_key. This function asks KMS to generate a random encryption key, and returns both the plaintext key as well as an encrypted version of that key - which is encrypted using a specified customer KMS master key. That KMS master key never leaves KMS.

That's a bit hard to follow so let's say it again. This function will provide you with a key you can use to encrypt data locally and then throw away. It also provides you with the same key which has been encrypted by a master key that is stored in KMS. I can transmit that encrypted key all day long, and without access to the KMS master key I used to generate it there is no value to it. It's just a blob of random data.

The other important and interesting point of the generate_data_key function is that I can provide it with an "EncryptionContext". This is a simple key/value pair I send in the function request. Without the matching information being provided on decryption time, the key can never be decrypted. Think of it as an additional context based password for the key, or something that you can use to tie the key to a particular usage or application.

A neat additional feature of the "EncryptionContext" is that every time the generated key is decrypted, the context is logged in AWS CloudTrail. This provides a built-in method to track the usage of your key, and any items that are encrypted with it.

And as for decrypting the key, I use the aptly named decrypt function. I send it the encrypted key, with the proper "EncryptionContext" values I used to generate it, and it figures out what customer Master key was used and decrypts the key, sending me back the original plaintext key. If I don't have the proper access to the right customer Master key, or I provide the wrong context values, all I will get is an "InvalidCiphertextException" error in response.

Great! So now I have a randomly generated key, and a mechanism for transmitting it and storing it securely. What can I do with it, you ask? Well for that let's take one more look at S3, particularly the Ruby SDK functions located under the "Encryption" section.

There exists (and to my knowledge this only applies to the Java and Ruby AWS SDKs) a special S3 client called the encryption client. This will perform envelope encryption on files uploaded to S3 with it, or to quote the documentation:

The goal of envelope encryption is to combine the performance of fast symmetric encryption while maintaining the secure key management that asymmetric keys provide.

A one-time-use symmetric key (envelope key) is generated client-side. This is used to encrypt the data client-side. This key is then encrypted by your master key and stored alongside your data in Amazon S3.

When accessing your encrypted data with the encryption client, the encrypted envelope key is retrieved and decrypted client-side with your master key. The envelope key is then used to decrypt the data client-side.

That's a lot of keys. But perhaps I can explain it like this: We are going to feed this encryption client the key we got from KMS. It is going to generate it's own key and encrypt the secret data we send. Then it will encrypt it's key with the one we sent it and store that in the metadata of the S3 file that contains our data. When we want to retrieve our secrets, we are going to provide the function with our original KMS key, and it's going to decrypt it's key with it, and then use that to decrypt the secret data.

And the cool part is, because our KMS key is encrypted as well, we're going to store it right there with the secrets file. When we want to get our secrets back, we are going to download our KMS key, decrypt it via KMS, and then send that key to the S3 encrypted client to use.

Whew! It sounds pretty complicated, but let's look at some code and you can probably see it better in action.

Encryption phase

This phase is usually done from a developer workstation to put the proper data in place.

Step one: request a new KMS key 

def self.fetch_new_key(app_context, master_key)
  kms_client = Aws::KMS::Client.new()
  genkey = kms_client.generate_data_key({ 
    key_id: master_key,
    key_spec: "AES_256",
    encryption_context: { 
      "Application" => app_context,
    }
  })
  return genkey.ciphertext_blob, genkey.plaintext
end 
In step one, I am requesting a new key from KMS. I am giving it a KMS master key id (master_key) and, because I am using this as part of CI/CD pipeline work for applications, I have decided to use an "EncryptionContext" composed of the key "Application" with a value that should be the name of the application that it is being used for. This could be any arbitrary key/value pair. This is going to return to me a plaintext key (plaintext), and an encrypted version of that key (ciphertext_blob).

 

Step two: open an S3 client connection

s3client = Aws::S3::Client.new()

 Easy enough. I am going to pass this client into the following sections so I only have to open a single client connection.

Step three: Upload the encrypted KMS key

def self.upload_key(s3client,newkeyblob,remote_filename,bucket,sse)
  keyfile_name= remote_filename+ ".key"
    newkeyblob64 = Base64.encode64(newkeyblob)
    if sse == "none"
      s3client.put_object({body: newkeyblob64,
                           key: keyfile_name,
                           bucket: bucket
      })
    else
    s3client.put_object({ 
      body: newkeyblob64,
      key: keyfile_name,
      bucket: bucket,
      server_side_encryption: sse
    })
  end
end

So here I am taking that encrypted key (ciphertext_blob) and saving it to S3 via a standard S3 client put_object under the name of "whatever my secrets file is called" plus ".key". I have to base64 encode the encrypted key here to have it stored properly in S3.

Note: the particular client we were using this for at the time had varying bucket policies that enforced S3 Server Side Encryption - sometimes with S3 managed keys, and sometimes with KMS managed keys. So as I wrote this gem I put the options in to do any of the SSE options, including none.

Step four: Encrypt and upload the secrets file

def self.upload_file(s3client,plaintext_key,local_filename,remote_filename,bucket,sse)
  begin
    filebody = File.new(local_filename)
    s3enc = Aws::S3::Encryption::Client.new(encryption_key: plaintext_key,
                                            client: s3client)
    if sse == "none"
      res = s3enc.put_object(bucket: bucket,
                             key: remote_filename,
                             body: filebody
      )
    else
      res = s3enc.put_object(bucket: bucket,
                             key: remote_filename,
                             server_side_encryption: sse,
                             body: filebody
      )
    end
  rescue Aws::S3::Errors::ServiceError => e
    puts "upload failed: #{e}"
  end
end

And now we use the S3 Encryption client to encrypt and upload the secrets file. I feed it the local and remote filenames of my secrets file, the bucket location, the SSE option, and most importantly - the plaintext key I generated in step 1. At no time is the plaintext key stored on the filesystem, it is only passed as a variable.

At this point, you will have an encrypted secrets file in S3 (for example "secrets.txt") and an encrypted key file in the same location ("secrets.txt.key").

Decryption phase

This phase is done on the final endpoints that are being configured.

Step one: open my S3 client connection

s3client = Aws::S3::Client.new()

Step two: get the encrypted KMS key file

def self.fetch_key(s3client,filename,bucket)
  keyfile_name= filename+ ".key"
  keyvalue=s3client.get_object(
    key: keyfile_name,
    bucket: bucket
  )
 keyval64 = Base64.decode64(keyvalue.body.read)
 return keyval64
end

Go get the encrypted key file and base64 decode it.


Step three: decrypt the KMS key

def self.decrypt_key(keyvalue,app_context)
  kms_client = Aws::KMS::Client.new()
  plainkey = kms_client.decrypt(
    ciphertext_blob: keyvalue,
    encryption_context: { 
      "Application" => app_context,
    }
  )
  return plainkey.plaintext
end

Here we are decrypting the key via a KMS call. KMS can figure out what customer Master key was used (as it's stored in the encrypted key), but note here that I am sending a context in again that must match the one I gave when I generated the key.

Step four: get and decrypt the secrets file

def self.fetch_file(s3client,plaintext_key,local_filename,remote_filename,bucket)
  begin
    s3enc = Aws::S3::Encryption::Client.new(encryption_key: plaintext_key,
                                            client: s3client)
    res = s3enc.get_object(bucket: bucket,
                           key: remote_filename,
                           response_target: local_filename)
  rescue Aws::S3::Errors::ServiceError => e
    puts "retrieval failed: #{e}"
  end
end


Finally we download the file located in the provided bucket under the remote filename, and decrypt it with our KMS key, saving it as the given local filename. The original file is now sitting on our local filesystem ready to be used.

And there you have it! Of course this can be used for more than CI/CD pipelines, but that's the purpose I originally created it for. I hope this post is informative, and that this can help you with your secrets management problems - as well as shining some light on the cooler features of KMS.

The source code is available on GitHub at https://github.com/DonMills/ruby-kms-s3-gem.The Ruby Gem is available on RubyGems at https://rubygems.org/gems/s3encrypt.

Learn more about our DevOps and Cloud service offering.

Don Mills
Don Mills
Cloud Security Architect
Contact Don

Related Articles