Manage federated workload identities with AWS IAM and Vault Enterprise

In this tutorial, you will learn how to configure and use AWS IAM OIDC identity providers to securely connect your Vault plugin.

Challenge

Plugins such as the AWS Secrets engine originally required static security credentials. The user supplies the long lived and highly privileged AWS credentials in the plugin configuration. For AWS this was the AWS_ACCESS_KEY and the AWS_SECRET_ACCESS_KEY.

$ vault write aws/config/root \
    access_key=<<AWS_ACCESS_KEY>> \
    secret_key=<<AWS_SECRET_KEY>> \
    region=us-east-1 

Some organizations have strict requirements and must use of short lived, ephemeral credentials. For such organizations, this configuration is not acceptable.

Vault provides root credential rotations API, but this still requires an initial set of long lived, highly privileged credentials credentials. Even the possibility of mishandling credentials during configuration might be unacceptable.

Solution

A identity provider (IdP) with a trust relationship with Vault ensures no use of static credentials.

With an identity provider, you can manage your user identities in Vault and give these temporary user identities permissions to use AWS resources.

This tutorial instructs the user how to set up this trust relationship between a cloud identity provider, Vault Enterprise, and a Vault plugin. Identity provider provides a method to securely configure a Vault plugin. Through an IAM role, Vault has access to specific capabilities and provides temporary security credentials to access AWS.

Personas

The intended audience of this tutorial is an Vault or AWS system administrator with privileged access and the ability to create AWS IAM identity providers.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Start interactive lab

Prerequisites

This tutorial requires an administrator access to an AWS account, Terraform, Vault CLI and the example Terraform configuration to create a demonstration environment.

• Vault Enterprise version 1.16 or later with premium license.
• Vault CLI
• AWS CLI
• Optionally - the use the Terraform CLI to create and configure the AWS OIDC identity provider.
• Amazon Web Services test account. This tutorial requires the creation of new cloud resources and but should incur minimal costs associated with the deployment and management of these resources. Your IAM user will need permission to create AWS IAM resources.
  • An AWS IAM user with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY used to create an AWS IAM Provider.
• Git
• ngrok installed and configured with an auth token.
• openSSL

Set up ngrok

Ngrok is a "reverse proxy". The set up in this tutorial exposes your dev Vault instance to the internet. Your local computer becomes a locally hosted server, anyone can access it through a subdomain of ngrok.com.

In this tutorial, ngrok provides a forwarding address that your local instance of Vault that will be accessible from AWS.

1. Open a terminal to run ngrok.

2. Follow the official instructions to download ngrok for your OS distribution and set up a free account: https://ngrok.com/download

3. After setting up your account and obtaining an authtoken, configure ngrok with the auth token:

   $ ngrok config add-authtoken <<token>>
   

4. Start ngrok.

   $ ngrok http 127.0.0.1:8200
   

   Example output:

   ngrok                                                                         (Ctrl+C to quit)
   
   New guides https://ngrok.com/docs/guides/site-to-site-apis/
   
   Session Status                online
   Account                       Hashicorp Account #1 (Plan: Enterprise)
   Update                        update available (version 3.10.1, Ctrl-U to update)
   Version                       3.9.0
   Region                        United States (us)
   Latency                       33ms
   Web Interface                 http://127.0.0.1:4040
   Forwarding                    https://XXNNNXXNNXXX.ngrok.app -> http://localhost:8200
   
   Connections                   ttl     opn     rt1     rt5     p50     p90
                                 0       0       0.00    0.00    0.00    0.00
   

5. Copy the ngrok Forwarding address for later use. This is the first part of the https://XXNNNXXNNXXX.ngrok.app -> http://localhost:8200 before the ->.

Vault setup

1. Open a second terminal and export an environment variable with a valid Vault Enterprise license.

   $ export VAULT_LICENSE=02MV4UU43BK5....
   

2. For the Vault to access the AWS Account, set AWS Access Key and Secret Access Key environment variables for the user you created for this tutorial.

   $ export AWS_ACCESS_KEY_ID=<<AWS_ACCESS_KEY>> && export AWS_SECRET_ACCESS_KEY=<<AWS_SECRET_ACCESS_KEY>>
   

3. Start a Vault dev server with the literal string root as the root token value, and temporarily setting the VAULT_API_ADDR to the ngrok forwarding address.

   $ VAULT_API_ADDR=<<NGROK_FORWARDING_ADDRESS>> vault server -dev -dev-root-token-id root
   

   The Vault dev server defaults to running at 127.0.0.1:8200, but through VAULT_API_ADDR ngrok provided a forwarding address that makes it available publicly. Now AWS IAM OIDC identity provider can access Vault on your local machine.

   Insecure operation

   Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.

   The Vault server is ready.

Confirm ngrok configuration

1. Open a third terminal.

2. In the new terminal, export an environment variable for the vault CLI to the ngrok forwarding address.

   $ export VAULT_ADDR=<<FORWARDING_ADDRESS_FROM_NGROK>>
   

   Setting the VAULT_ADDR to the ngrok forwarding address forces the Vault CLI to use ngrok.com which forward to your local instance of Vault.

3. Export an environment variable for the vault CLI to authenticate with the Vault server.

   $ export VAULT_TOKEN=root
   

4. To validate the correct set up of ngrok and Vault, check the status of Vault.

   $ vault status
   Key             Value
   ---             -----
   Seal Type       shamir
   Initialized     true
   Sealed          false
   Total Shares    1
   Threshold       1
   Version         1.16.2
   Build Date      2024-04-22T16:25:54Z
   Storage Type    inmem
   Cluster Name    vault-cluster-beeda372
   Cluster ID      cf3c7383-ff3a-3502-964c-9ef3d2f094f4
   HA Enabled      false
   

   A similar response confirms that ngrok is forwarding to your Vault instance.

Obtain thumbprint for intermediate certificate authority

During configuration, Vault requires a thumbprint from the top intermediate certificate authority that signed the certificate.

A certificate thumbprint is an unique identifier used to verify the certificate.

To get the thumbprint, ether follow the directions at Obtain the thumbprint for an OpenID Connect identity provider and copy down the thumbprint for use in this tutorial, or you can follow the steps below to retrieve the thumbprint.

1. Extract the domain from the URI of the JSON web key set of the IdP.

   $ export DOMAIN=$(curl -s http://127.0.0.1:8200/v1/identity/oidc/provider/default/.well-known/openid-configuration | jq -r '.jwks_uri | sub("^https://";"") | sub("\/(.*)";"")')
   

2. Using the DOMAIN from above, this command extracts the certificate and calculates thumbprint.

   $ export THUMBPRINT=$(openssl x509 -in <(openssl s_client -servername "keys.$DOMAIN" -showcerts -connect "keys.$DOMAIN:443" 2>/dev/null) -outform DER| sha1sum -z | awk '{print $1;}')
   

Create AWS OIDC identity provider

This tutorial provides two methods to set up the AWS open identity provider, AWS CLI and Terraform.

1. In a new terminal window, clone the repository at learn-vault-plugin-wif.

   $ git clone https://github.com/hashicorp-education/learn-vault-plugin-wif
   

2. Move into the cloned repository.

   $ cd  learn-vault-plugin-wif
   

3. For the Vault CLI to access the AWS Account, set AWS Access Key and Secret Access Key environment variables.

   $ export AWS_ACCESS_KEY_ID=<<AWS_ACCESS_KEY>> && export AWS_SECRET_ACCESS_KEY=<<AWS_SECRET_ACCESS_KEY>>
   

4. Using the ngrok forwarding address copied earlier, update the code block with your ISSUER and paste in the terminal. This is the

   $ export ISSUER=<<ISSUER>>
   

   Note: No not include the "https://" when setting this environment variable.

5. Using the fingerprint created during the prerequisites, create another TF_VAR environment variable.

   $ export THUMBPRINT=<<THUMBPRINT>>
   

Set up AWS secrets engine

1. Using the ngrok forwarding address copied earlier, update the following code block and paste in your terminal.

   $ export ISSUER=<<ISSUER>>
   

   Note: No not include the "https://" when setting this environment variable.

2. In a new terminal, set some Vault environment variables.

   $ export VAULT_TOKEN=root && export VAULT_ADDR="https://$ISSUER"
   

3. Configure the identity provider issuer.

   $ vault write identity/oidc/config issuer=$VAULT_ADDR
   WARNING! The following warnings were returned from Vault:
   
   * If "issuer" is set explicitly, all tokens must be validated against that
   address, including those issued by secondary clusters. Setting issuer to
   "" will restore the default behavior of using the cluster's api_addr as the
   issuer.
   

4. Enable the AWS Secrets engine.

   $ vault secrets enable aws
   Success! Enabled the aws secrets engine at: aws/
   

5. Configure the AWS secrets engine with a role allowing access to the AWS identity provider and the audience intended.

   $ vault write aws/config/root \
      identity_token_audience="${TOKEN_AUDIENCE}" \
      role_arn="${AWS_ROLE_ARN}"
   

6. The AWS IAM user my-role with a policy document with to get the current user information.

   $ vault write aws/roles/my-role \
      credential_type=iam_user \
      policy_document=-<<EOF
   {
      "Version": "2012-10-17",
      "Statement": [
         {
            "Effect": "Allow",
            "Action": "iam:GetUser",
            "Resource": "arn:aws:iam::${AWS_ACCOUNT_ID}:user/vault-token-my-role-*"
         }
      ]
   }
   EOF
   

   Example output:

   Success! Data written to: aws/roles/my-role
   

7. Get AWS credentials from Vault.

   $ vault read aws/creds/my-role
   ---                -----
   lease_id           aws/creds/my-role/Ws0fpy9DJAgusZUQt0BH0EeS
   lease_duration     768h
   lease_renewable    true
   access_key         AKIA...
   secret_key         NN...
   session_token      <nil>
   

   This creates a temporary user with my-role attached to it. The policy attached allows the IAM user to identify itself.

8. Use the access_key and secrets_key above to reset your AWS environment variables.

   $ export AWS_ACCESS_KEY_ID=<<AWS_ACCESS_KEY>> && export AWS_SECRET_ACCESS_KEY=<<AWS_SECRET_ACCESS_KEY>>
   

9. Use the AWS CLI to check which user you are using the credentials of.

   $ aws iam get-user
   {
      "User": {
         "Path": "/",
         "UserName": "vault-token-my-role-1718035748-ITpjTB8nP7mQiaygqWLa",
         "UserId": "AIDA...",
         "Arn": "arn:aws:iam::NNNNNNNNNNNN:user/vault-token-my-role-1718035748-ITpjTB8nP7mQiaygqWLa",
         "CreateDate": "2024-06-10T16:09:08+00:00"
      }
   }
   

Set up AWS authentication method

In addition to using an identity provider to configure the AWS secrets engine, the AWS authentication method can use an identity provider instead of static credentials for set up.

The last section included the IAM policies needed to set up the AWS authentication method, but the Vault configuration for the AWS authentication method is still required.

1. In the same terminal used in the last section, write the OIDC configuration.

   $ vault write identity/oidc/config issuer="${VAULT_ADDR}"
   

   Example output:

   WARNING! The following warnings were returned from Vault:
   
   * If "issuer" is set explicitly, all tokens must be validated against that
   address, including those issued by secondary clusters. Setting issuer to
   "" will restore the default behavior of using the cluster's api_addr as the
   issuer.
   

2. Enable AWS authentication.

   $ vault auth enable aws
   Success! Enabled aws auth method at: aws/
   

3. Configure the AWS authentication to use the identity provider.

   $ vault write auth/aws/config/client \
      identity_token_audience="${TOKEN_AUDIENCE}" \
      role_arn="${AWS_ROLE_ARN}"
   

   Example output:

   Success! Data written to: auth/aws/config/client
   

4. Attach dev-role-iam to the token provided by the AWS auth method.

   $ vault write auth/aws/role/dev-role-iam \
      auth_type=iam \
      bound_iam_principal_arn="arn:aws:iam::${AWS_ACCOUNT_ID}:user/*"
   

   Example output:

   Success! Data written to: auth/aws/role/dev-role-iam
   

5. Open a new terminal, and set the Vault environment variables.

   $ export VAULT_TOKEN=root && export VAULT_ADDR=<<FORWARDING_ADDRESS_FROM_NGROK>>
   

6. For the Vault CLI to access the AWS Account, set AWS Access Key and Secret Access Key environment variables.

   $ export AWS_ACCESS_KEY_ID=<<AWS_ACCESS_KEY>> && export AWS_SECRET_ACCESS_KEY=<<AWS_SECRET_ACCESS_KEY>>
   

7. Authenticate with Vault using dev-role-iam.

   $ vault login -method=aws role=dev-role-iam
   
   Success! You are now authenticated. The token information displayed below
   is already stored in the token helper. You do NOT need to run "vault login"
   again. Future Vault requests will automatically use this token.
   
   Key                      Value
   ---                      -----
   token                    hvs...
   token_duration           768h
   token_renewable          true
   token_policies           ["default"]
   identity_policies        []
   policies                 ["default"]
   token_meta_role_id       4f39f8dc-1587-1b3b-cba7-5558b371bbb5
   token_meta_account_id    794723625284
   token_meta_auth_type     iam
   

   You have authenticated with Vault using the AWS auth method from an identity provider.

   The output displays details associated with the Vault role.

Clean up

1. In the terminal with Vault, press Ctrl + C to stop the running Vault dev server.

2. In the terminal with ngrok, press Ctrl + C to stop the running server.

3. Close all open terminals.

Additional discussion

This tutorial used ngrok as a reverse proxy to make the Vault dev instance publicly routable. However, this is undesirable for most production use-cases.

For production, we recommend using a proxy to enable access to only the API endpoints strictly necessary.

In the case of workload identity federation, these are:

- ${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/identity/oidc/plugins/.well-known/openid-configuration
- ${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/identity/oidc/plugins/.well-known/keys

Substitute in appropriate values for your VAULT_ADDR and VAULT_NAMESPACE.

Be sure to also update the issuer in identity/oidc/config.

Where possible, consider restricting access to the proxy to AWS.

Next steps

Organizations in a highly regulated industry have rules against the use of static credentials, and until plugin Workload Identity Federation, plugin set up required static credentials.

In this tutorial, you learned how to set up AWS OIDC identity provider and avoided any use of static credentials in the process. This was possible through a trust relationship between Vault and AWS OIDC identity provider. Vault will manage the identities on the cloud provider and give these temporary user identities permission to use AWS resources.

For more information, see individual plugin documentation for instructions on how to set up workload identity federation.

• AWS Auth Methods - Plugin Workload Identity Federation
• AWS Secrets Engine - Plugin Workload Identity Federation
• Azure Auth Methods - Workload Identity Federation
• Azure Secrets Engine - Plugin Workload Identity Federation
• Google Cloud Auth Methods - Workload Identity Federation
• Google Cloud Secrets Engine - Plugin Workload Identity Secrets Engine