Introducing: CloudTracker, an AWS CloudTrail Log Analyzer
Today we are pleased to announce a new open-source tool from Duo Security for easily analyzing CloudTrail logs from Amazon Web Services (AWS)!
In order to implement the Principle of Least Privileges for our IAM (Identity and Access Management) users and roles in our AWS accounts, we wanted to ensure the IAM privileges these actors were granted were actually being used. Any privileges that have been granted but not used are opportunities to prune down the privileges.
CloudTracker reviews CloudTrail logs to identify the API calls made by an actor and compares this with the IAM privileges that the actor has been granted to identify privileges that can be removed. Check out our CloudTracker tool on Github.
One of the driving motivators for this tool was the realization that as a user assumes roles in an account, or across accounts, it becomes much more tedious to identify what actions they've taken. Amazon advises AWS customers use a multi-account strategy to implement security segmentation, reduce the blast radius in incidents, manage billing, and more.
Duo adheres to this guidance and has many AWS accounts. Duo employees that are allowed to access accounts often do so through cross-account roles, so we had a compelling need to build a tool that could help us identify any unused privileges that had been granted.
For example, imagine you have two users Alice and Bob that use an "admin" role. Their user privileges grant them read access in the account and the ability to assume this "admin" role. Alice uses the privileges granted by this role heavily, creating new EC2 instances, new IAM roles, and all sorts of actions, whereas Bob only uses the privileges granted by this role for one or two specific API calls.
If you looked at what actions were taken by this role, you'd find it was well-used due to Alice, but you wouldn't be able to easily see that Bob was over-privileged. If you want to identify only the actions taken by Bob as the "admin" role, you need to identify every "AssumeRole" call he has made, and then for each session, analyze what was done. This is where a tool was needed to make analysis less tedious and manual.
CloudTrail Logs
CloudTracker uses AWS CloudTrail logs and IAM policy information for an account. CloudTrail records the API calls made in an account, but does have limitations. The most significant is data level actions are not recorded in CloudTrail, such as S3 object access. CloudTrail can be configured to log some of these data level activities, but there are still some AWS API calls that are never recorded in CloudTrail; therefore, CloudTracker can't determine if a user with these privileges is over-privileged.
Instead of using CloudTrail logs, an alternative solution would have used AWS's Access Advisor service, but that has a number of limitations that caused us to focus on CloudTrail logs instead. First, there are no API calls to collect information from Access Advisor. The only way of viewing Access Advisor information is through the web UI. Netflix uses Access Advisor via their Aardvark and RepoKid tools and works around this limitation by using PhantomJS to log in and scrape the data.
Another major limitation with Access Advisor is that you can't trace individual users as they assume roles within an account or across accounts. Finally, the information available from Access Advisor is not very granular. For example, Access Advisor cannot tell you how often a privilege was used, what resources were acted on, or the specific API call used. CloudTracker does not currently take resources into account either, but because its source of truth are CloudTrail logs, it can be modified in the future to display more detailed information and offer tighter privilege restriction advice based on resource attributes.
Mozilla Hindsight and ElasticSearch
CloudTracker requires you to have loaded your CloudTrail logs into ElasticSearch. There are tutorials available on configuring LogStash or other tools to monitor an S3 bucket and continuously feed CloudTrail logs into ElasticSearch, but we realize that not everyone has this setup or wishes to run and maintain a full-time ElasticSearch cluster. You may wish to only use CloudTracker once a quarter and so you'll need to spin up an ElasticSearch cluster, feed in your data, run CloudTracker, and then shut down the cluster.
Unfortunately, many log ingestion tools were made for trailing logs, and not for this use case of quickly ingesting many gigabytes of logs at once. After trialing many of the existing tools for ingesting logs into ElasticSearch, we found Mozilla's Hindsight tool to be the best for this use case. Hindsight is based on lessons learned from Mozilla's previous log ingestion tool, Heka. Along with the release of CloudTrail, we've included in our repo instructions for ingesting a log archive into ElasticSearch using Hindsight.
Ingesting logs into ElasticSearch with Hindsight can still take hours, but once the logs have been loaded into ElasticSearch, running CloudTracker against it takes seconds.
CloudTracker Use Case Examples
From our scenario earlier, let's use CloudTracker against an account with two users, "alice" and "bob" that can each use an "admin" role, and that each have read privileges in the account without assuming a role. Looking at some of the output of CloudTracker, we can see the privileges granted to Alice and whether she has used them or not:
python cloudtracker.py --account demo --user alice
...
cloudwatch:describealarmhistory
cloudwatch:describealarms
- cloudwatch:describealarmsformetric
- cloudwatch:getdashboard
? cloudwatch:getmetricdata
...
This shows that Alice has used some CloudWatch privileges (ex. DescribeAlarms), has not used others (ex. GetDashboard), and it is unknown whether or not she has made calls to GetMetricData because that is not recorded in CloudTrail.
If we now look at the "admin" role, and filter on only the commands that have been used, we can see that there are two calls that have been made with it.
python cloudtracker.py --account demo --role admin --show-used
Getting info for role mfa_admin
s3:createbucket
iam:createuser
Looking at the calls made by Alice as an "admin", we see she has used these two calls:
python cloudtracker.py --account demo --user alice --destrole admin --show-used
Getting info on alice, user created 2017-09-01T01:01:01Z
Getting info for AssumeRole into admin
s3:createbucket
iam:createuser
But if we look at Bob when he assumed the "admin" role, we can see he only made an S3 call to CreateBucket, and therefore we might want to remove some of the IAM service privileges from him:
python cloudtracker.py --account demo --user bob --destrole admin --show-used
Getting info on bob, user created 2017-10-01T01:01:01Z
Getting info for AssumeRole into admin
s3:createbucket
Stay in touch!
If you’re interested in the intersection between security and running a highly-available service on AWS, please contact Duo's Production Engineering team at prodeng-ext@duo.com.