I found myself in a real head-scratcher when I noticed one of our clients was spending $8,000 every single month on AWS CloudTrail.
What is AWS CloudTrail?
If you’re not familiar with AWS CloudTrail, it’s a nifty service that helps you enable governance, compliance, and operational and risk auditing of your AWS account. When an action is taken by a user, role, or an AWS service, it’s recorded as an “event” in CloudTrail. Due to regulatory compliance requirements, many of our clients leverage Security Information and Event Management (SIEM) SaaS products to monitor their CloudTrail event logs for suspicious or unauthorized activity. These tools help organizations monitor their cloud account activity and better manage security by prioritizing the security events, generating alerts, and retaining the security data for auditing purposes.
We were asked to review and optimize AWS spending
We were brought in by this client to support product development while working with the existing development teams to adhere to best practices around development/deployment. We hold weekly meetings with this client to review their AWS spending and determine what we can possibly optimize. Usually, these discussions revolve around determining the right sizes for EC2 instances and identifying old resources that are no longer in use.
Surprisingly, one of their cost charts showed the huge monthly spend on CloudTrail. This isn’t the first time I’ve encountered a client paying way too much for this service, so I decided to dig in to figure out what was going on.
What I discovered...
I discovered our client previously provisioned multiple CloudTrails to support POCs for different SIEM SaaS offerings over the years, such as Alien Vault, Expel Workbench, and Lacework. Each AWS account in their organization contained at least 2-3 active CloudTrail Trails, capturing duplicate copies of the same events to support the different SIEMs. When I tracked down where all these Trails were coming from, it was clear each SIEM provided its own CloudTrail Infrastructure as Code (IaC) to bootstrap the AWS Account for use with their product.
This might sound okay at first since the IaC provided by the SIEM gives the customer a quick and easy way to get started using the tool. However, CloudTrail’s pricing model can be punishing for AWS accounts with multiple CloudTrail Trails. This is because AWS only gives you the first copy of each account’s management events for free. Additional copies are charged at $2 per 100,000 events. That $2 adds up quickly when hundreds of millions of events happen each month. So instead of allowing each SIEM to create its own CloudTrail, we could cut costs using our own custom IaC to build a single CloudTrail in each account that can be used to support the various SIEMs.
A relatively simple solution
I let our client know what I found and proposed a quick and relatively simple solution. I switched the SaaS offerings over to an AWS Organizations consolidated trail that funneled all the account’s CloudTrail logs into a bucket in the Main account. Some of the services wanted differing event formats, so we needed an intermediary Lambda Function to translate the events into the desired formats and send them out to the respective SaaS SQS queues. Once the SQS queues were receiving duplicate copies of each event (one from the individual account and one from the Main account), we were able to disable the Trails created by the SaaS IaCs in the individual accounts and stop capturing multiple copies of each event and paying such a high monthly fee.
And that, my friends, is how we saved our client $92,000 a year.
Ready to cut costs?
If your organization uses AWS CloudTrail and you want to determine if you too can cut costs, here are a few suggestions on where to start:
- Use AWS Budgets to set custom budgets that alert you when costs or usage exceed the budgeted amount.
- Investigate the use of multiple trails. Note that when using AWS Organizations to create an Organizations Trail, AWS Organizations will create a Trail in each member account. Additional charges will be incurred if an individual member account creates an additional trail to collect the same events as the Organizations trail.
- Consider leveraging the “Fanout S3 Event Notifications to Multiple Endpoints” design pattern to forward CloudTrail event notifications to multiple destinations when the event processors require differing event formats. We used the “Alternative Architecture” with a ‘master’ Lambda function to process, translate, and forward the events to the respective SIEM event ingestion queues.
