Knowing when credentials have been compromised or being able to identify unauthorized login attempts is a difficult security challenge, cloud systems or not. A new tool from Netflix’s security tools and operations team helps manage these challenges in the cloud.
Netflix relies on Amazon Web Services for its infrastructure and computing needs, and needs to know when a credential is potentially compromised, Will Bengtson, a senior software security engineer at Netflix, said at Black Hat USA. Netflix has hundreds of thousands of virtual server instances on AWS and utilizes AWS Security Token Service to generate credentials for AWS Identity and Access Management.
Netflix wanted to make sure tokens from STS weren’t being abused. IT administrators generate tokens with AWS STS to grant trusted users temporary and limited access to specific resources. The STS receives authentication information from AWS IAM and generates credentials with expiration windows that can be as short as a few minutes or as long as a several hours. While AWS won’t give access to expired credentials, users can request new credentials before the old one expires, and the service dynamically generates credentials as needed. By using STS, IT administrators don’t have to manage credentials within AWS IAM for these short-term requests.
At Netflix, we have hundreds of thousands of servers. They change constantly, and there are 4,000 or so deployments every day," Bengtson said. "I really wanted to know when a credential was being used outside of Netflix, not just AWS.
If credentials were compromised, an unauthorized users could set up new AWS infrastructure and log in using those compromised credentials.
Amazon offers a number of tools on AWS to continuously scan for threats and to log every activity across instances. For example, GuardDuty uses machine learning to analyze information collected in AWS CloudTrail, VPC Flow Logs, and AWS DNS logs to find anomalies and alert customers. Netflix tracks event history for AWS account activity using CloudTrail, the logging service enabled by default on AWS instances that tells administrators everything that is happening on the system.
Bengston created Trailblazer, an open-source tool currently available on GitHub, to look at which AWS API calls were logged by CloudTrail and to make it easier to find specific functions. CloudTrail logs are by definition huge, so searching line-by-line is not sustainable, especially considering the size of Netflix’s infrastructure. While Netflix uses Trailblazer to find compromised STS credentials, it can be used to with other temporary security credentials such as ECS and EKS, Bengston said.
Trailblazer collects the first API call from the IP address, instance ID, assumed role records, and other related AWS data.
The tool "will enumerate all of your API calls in your environment and associate that log with what is actually logged in CloudTrail," Bengtson said. "Not only are you seeing that it's logged, you're seeing what it's logged as."
While it’s possible to find unauthorized logins by comparing each IP address in CloudTrail against a list of all IP addresses the organization owns, it is not a sustainable task. Maintaining such a list is a challenge, especially considering that the ephemeral nature of the cloud means IP addresses change over time. This approach is not sustainable for any decent-sized AWS environment, let alone the operation Netflix has.
The administrator using Trailblazer doesn’t need to have a list of all IP addresses or know how the infrastructure is organized. By looking at what calls are being made, the administrator can quickly determine whether the calls for those AWS credentials were coming from outside the organization.
The first call wins," Bengtson said. "As we see the first use of that temporary [session] credential, we're going to grab that IP address and log it.
For example, an attacker would call the “GetCallerIdentity” function to find out what account the attacker is using. A user using legitimate credentials would already know what account is being used and not need to call the function.
The methodology behind Trailblazer relies heavily on how AssumeRole calls are logged by CloudTrail. The tool scales with the the AWS environment and number of accounts and doesn’t require administrators to maintain a full list of IP addresses allocated to the account. However, it should not be viewed as the only security tool.
“Do bear in mind the “defense in depth” truism: this should only constitute one “layer” of your security tactics in AWS,” Bengtson wrote.