AWS Common Issues – Part 2

If you ever developed any web application with a certain degree of complexity you must have realized that at some point you also need to store credentials, tokens, secrets, and sensitive information on these applications.

In an AWS environment, these applications usually run in EC2 instances, which are virtual machines created by Amazon under a set of parameters we get to decide. In other cases, they could be running in containers, or have been created by Kubernetes, and this would open up a wide range of attacks that will be left for another white paper. For now, let’s analyze how to secure application secret’s in EC2 instances.

First, let’s take a look at the main ways attackers use to access sensitive information in this context:

1- Root access: A root access to an AWS account implies unrestricted access to the information in it. The Root user in Linux systems grants the same level of privileges and access. If an attacker is Root, whether in linux or AWS, then it is game over.

2 – Bad practices when storing sensitive information:  Using public URLs, public objects in S3 buckets, or hardcoding credentials in the application source code, all imply easy ways for attackers to access our data.

3 – Privilege escalation: Here lies every technique of privilege escalation mentioned in our last edition (“Top 5 security tips: AWS Cloud Infrastructure”), in the “Principle of least privilege” section. If an escalation technique is possible, any attacker with low privileges will have access to EC2 instances and will be able to read any information they contain.

4 – User-Data: AWS gives us the option of using the user-data field, available in the AWS metadata API, through the URL http://169.254.169.254/1.0/user-data. If we place secrets there, any application that opens a socket in that direction will have access to our sensitive information. This includes, of course, attackers exploiting an SSRF bug in the application, erroneous Firewall and/or reverse proxy configurations, and attackers with limited AWS access, but with the privilege to read EC2 metadata.

These are some of the possible attack vectors.

How can we avoid them? There are several options available, but none of them is a golden solution: each has advantages and disadvantages and requires further analysis:

1. Using IAM roles.

2. Hardcoding the secret in the AMI image.

3. Hardcoding the secret in a public/authenticated URL.

4. Using private S3 objects and providing the URL yourself.

5. Sending secrets in the instance’s deployment time.

6. Sending the secret through user-data.

7. Using AWS Secrets Manager.

8. Using an identity-based secret and encryption management system.

As we can see, there are multiple options. Each of them is useful for a specific context.However, it would be nice to have some general solutions. In this article, we’ll see solutions we maynapply in most cases with the least attack vectors possible.With this concept in mind, we will provide two technical recommendations that can be used to share secrets in EC2 instances.The first technique to share AWS services secrets will allow us not to store secrets at any point.We will simply use the IAM role system in conjunction with the AWS EC2 Metadata API to gain access and perform the required actions on the services we need.

If we need to provide access to the whole range of services Amazon AWS provides, the best option in terms of security is -without a doubt- to use the Roles system that IAM offers.

Instead of sending or storing Access Key and Secrets Key that will later be used by SDKs or Cli-mode interfaces to access and operate with the services we want, we have the option of enabling temporary credentials. They change after a couple of minutes and can be used by SDKs in a native way or through CLI interfaces, without the need of exposing a specific user’s permanent credentials or access keys.

The advantages of this technique are:

-Native support in AWS SDKs.

-Once the roles are set and implemented in the code, everything else is managed by AWS.

-It provides temporary credentials, so if the EC2 instance is compromised and these temporary credentials are leaked, the attackers will only have access to the role’s services for a couple of minutes. After that, they will need to obtain a new temporary credential.

– It allows a more granular management of the granted privileges and a better control/audit of them, promoting the “least privilege” principle.

If you’d like to know how to implement IAM roles in EC2 instances at a technical level, check out these links:

A Safer Way to Distribute AWS Credentials to EC2 | AWS Security Blog

IAM roles for EC2 instances – Simplified Secure Access to AWS service APIs from EC2 | AWS News Blog

Let’s talk about what happens if we need to provide our EC2 instances with secrets that are not AWS-native, such as:

-Database credentials.

-Application credentials.

-OAuth tokens.

-Application Programming Interface (API) keys.

Usually, when we develop an application that needs information from a database or an API, credentials or secrets are hard-coded in order to access that information in the application itself, exposing the source code to potential leaks, revealing the secrets and obstructing their rotation. In order to rotate the secret, we would need to update it in every used instance in our app. Because of this, the secret rotation practice usually found in audited applications is very low or practically zero.

By using AWS Secrets Manager, we can store and safely supply these secrets in our EC2 instances or any script/application that requires it by using an API call to Secrets Manager in order to obtain that secret in a programmatic way.

In addition to keeping our code free from secrets, it automatically helps us by rotating them according to a set of rules we may define, reducing the risks of compromised information.

One of the best features of this solution is its integration with AWS Identity and Access Management (IAM), which allows us to attach permission policies to users, groups, and roles in order to allow/deny access to specific secrets. This way, we can have precise and granular control on who has access to what, reducing the surface exposure to attacks and/or leaks.

If you’d like to know how to implement Secrets Manager in EC2 instances at a technical level, check out these links:

AWS Secrets Manager

Move hardcoded secrets to AWS Secrets Manager

Set up single user rotation for AWS Secrets Manager