How to store your .env files in AWS Secrets Manager

Table of Contents

1. How to properly store your .env files in AWS Secrets Manager
2. Migrating your .env files to AWS
3. Frequently asked questions
4. Conclusion

How to properly store your .env files in AWS Secrets Manager

You probably don't check out your .env files to your code repository because they contain secrets, so to share these values you have to use different methods such as sending it via email, Slack, one-time links, etc. Moving all your secrets to AWS will help you centralize and simplify the management and lifecycle.

This way you would have all environment keys/variables (or .env files) in one place that is secure and only accessible to those who have the right credentials.

Migrating your .env files to AWS

The idea sounds great, but now you have to migrate multiple .env files to AWS.

We have two possible scenarios for the .env files:

1. We only need to store some values that are going to be used as-is across multiple environments
2. We need to store values depending on each environment (ie, backend endpoint, database credentials, etc.)

(1) For any project with no difference between environments:

For this case, we can directly store the file as binary and re-generate the .env file for our local development, CI/CD pipelines using the following commands.

# Create
aws secretsmanager create-secret --name my-file-secret --secret-binary fileb://.env

# Get
aws secretsmanager get-secret-value --secret-id my-file-secret  --query SecretBinary --output text | base64 --decode > .env

Add the get line into your project's README.md file, include this step in your CI/CD pipeline and that's it.

(2) Projects where we need to have different values depending on the environment

First, we need to define a naming convention for our secrets which allows us to identify them via application/service name, environment and the key itself.

For a given service we can use:

• Service name
• Environment
• The secret we are storing

Giving us something like: <service-name>/<env>/<secret-name>/. For instance: user-api/dev/database-host.

This is going to be a bit trickier so let's write some bash to automate this process. We can use this bash script to iterate over the .env file and store each key individually.

#!/bin/bash

service_name="user-api" # Change per service/app
env="dev" # Change per env

while IFS="" read -r p || [ -n "$p" ]
do
  key=${p%%=*}
  value=${p#*=}
  key=${p%%=*}
  key_name=`echo "$key" |  tr '[:upper:]' '[:lower:]'` # Converting it to lower case
  value=${p#*=}
  secret_id="$service_name/$env/$key_name"

  echo "Storing: $secret_id" # Save this for later :) 
  
  aws secretsmanager put-secret-value \
  --secret-id $secret_id \
  --secret-string $value

done < .env

Once we have these values stored, we need to be able to retrieve them to re-generate our .env file Call this as: sh env.sh dev

#!/bin/bash

service_name="user-api"

set -euo pipefail
trap "exit 4" TERM
TOP_PID=$$
function die {
    echo "FATAL: $@"
    kill -s TERM $TOP_PID
}

function get_ssm_parameter {
    local out=$1
    local ssm_key=$2
    echo "... looking up SSM parameter $ssm_key ..."
    local r=$(aws ssm get-parameter --name "$ssm_key" --query Parameter.Value --output text --region us-east-1)
    [ -z "$r" ] && die "Unable to read SSM parameter $ssm_key"
    echo "Read $ssm_key from SSM parameter"
    eval "$out='$r'"
}

if [ $# -ne 1 ]; then
    echo "\
    Usage:
      env.sh <env>
    Available env:
    - dev
    - prod
    "
    exit 1
fi
env=$1
case $env in
    dev)
        echo "## Generating $env .env file ##"
        ;;
    prod)
        echo "## Generating $env .env file ##"
        ;;
    *)
        die "Unknown env id: $env"
        ;;
esac

# We need to manually define this for each project
database_host_key="$service_name/$env/database-host"
database_password_key="$service_name/$env/database-password"

get_ssm_parameter database_host $database_host_key
get_ssm_parameter database_password $database_password_key

cat <<EOT >> .env
DATABASE_PORT=3306 # Example of values that won't change
DATABASE_NAME=user-db
DATABASE_HOST=$database_host
DATABASE_PASSWORD=$database_password
EOT

Frequently asked questions

• How does storing .env files in AWS Secrets Manager compare to other methods like environment variables or encrypted files?

Storing .env files in AWS Secrets Manager offers several advantages over traditional methods like storing secrets in environment variables or encrypted files. Unlike environment variables, AWS Secrets Manager provides a centralized and secure repository for managing secrets, reducing the risk of accidental exposure. Additionally, AWS Secrets Manager offers features like automatic rotation of secrets, fine-grained access control, and integration with AWS services, making it a more robust solution for managing sensitive information compared to encrypted files or manual handling of environment variables.

• Can you elaborate on the security measures implemented by AWS Secrets Manager to ensure the confidentiality of stored secrets?

AWS Secrets Manager implements multiple layers of security to ensure the confidentiality of stored secrets. This includes encryption-at-rest using AWS Key Management Service (KMS) keys, encryption-in-transit using TLS, and strict access controls using AWS Identity and Access Management (IAM) policies. Secrets are encrypted before being stored in AWS Secrets Manager, and access to secrets is restricted based on IAM policies, ensuring that only authorized users or roles can retrieve or modify secrets.

• Are there any best practices or guidelines for managing permissions and access control to these secrets within AWS Secrets Manager?

When managing permissions and access control for secrets within AWS Secrets Manager, it's essential to follow the principle of least privilege. This means granting only the minimum level of permissions required for users or roles to perform their tasks. IAM policies can be used to define granular permissions, such as restricting access to specific secrets or allowing only read or write operations. Additionally, AWS Secrets Manager supports resource-based policies, which allow fine-grained control over who can access or modify individual secrets.

• In the second scenario mentioned, where different values are needed depending on the environment, how does the provided bash script handle cases where a key-value pair needs to be updated or removed?

In the second scenario where different values are needed depending on the environment, the provided bash script handles updates or removals of key-value pairs by using the put-secret-value operation of the AWS CLI. This operation updates the value of an existing secret or creates a new secret if it doesn't exist. If a key-value pair needs to be removed, the script can be modified to delete the corresponding secret using the delete-secret operation. By iterating over the .env file and storing each key-value pair as a separate secret in AWS Secrets Manager, the script ensures that secrets can be managed individually and updated as needed.

• Could you explain how the solution presented in the blog post integrates with existing CI/CD pipelines and deployment processes, especially in a team environment with multiple developers working on the same project?

The solution presented in the blog post can be integrated into existing CI/CD pipelines and deployment processes by incorporating the provided bash scripts into the pipeline configuration. For example, the script for migrating .env files to AWS Secrets Manager can be included as a step in the deployment process, ensuring that secrets are securely stored before deploying the application. Similarly, the script for generating .env files from secrets stored in AWS Secrets Manager can be executed during the deployment process to retrieve the latest secrets and configure the application environment. By integrating these scripts into CI/CD pipelines, teams can automate the process of managing secrets and ensure consistent deployment across different environments.

Conclusion:

1. Storing our env vars in AWS allows us to share them in a secure and convenient way
2. Only the allowed user/roles will have access to these values
3. Reduce the risk of checking out .env files or sharing them via chat/email
4. If any value gets compromised we have a central location to update the values and act faster