aws-amplify / aws-amplify/amplify-cli
consider writing function secrets directly to Lambda env vars
- Dominant language
- TypeScript
- Stars
- 2.9k
- Forks
- 825
- Avg merge
- 11d 23h
- Merged PRs (30d)
- 2
Description
### Is this feature request related to a new or existing Amplify category?
function, New category
### Is this related to another service?
_No response_
### Describe the feature you'd like to request
Currently when adding secrets to functions the CLI adds a snippet demonstrating how to fetch secrets from SSM, however this can have unintended side-effects for response times.
For this example we have two API routes:
1. `/hello` has one environment variable `NAME` with the value of `World`
2. `/hello-secrets` has one _secret_ `NAME` with the value of `World`
/hello-secrets handler
**NOTE**: this handler is using a revised SSM snippet for use with ESM, which also overwrites the env var holding the SSM key with the decrypted value
```js
import { SSMClient, GetParametersCommand } from '@aws-sdk/client-ssm'
const SECRET_NAMES = ['NAME']
async function getSecrets(secretNames) {
try {
const client = new SSMClient({ region: process.env.AWS_REGION })
const Names = secretNames.map((secretName) => process.env[secretName])
const command = new GetParametersCommand({ Names, WithDecryption: true })
const params = await client.send(command)
if (!params.Parameters.length || params.InvalidParameters.length) {
throw new Error('Unable to retrieve secrets from ParameterStore')
}
return params.Parameters?.reduce((acc, { Name, Value }, index) => {
if (Name)
acc[secretNames.find((secretName) => Name.endsWith(secretName))] = Value
return acc
}, {})
} catch (error) {
console.error(error)
throw new Error('Unable to get secrets', error)
}
}
async function loadSecrets(secretNames = SECRET_NAMES) {
console.log('Loading secrets')
try {
const secrets = await getSecrets(secretNames)
for (let [secretName, secretValue] of Object.entries(secrets)) {
process.env[secretName] = secretValue
}
console.log('Secrets loaded successfully')
return true
} catch (error) {
console.error(error)
throw new Error('Unable to load secrets', error)
}
}
let secretsLoaded = false
/**
* @type {import('@types/aws-lambda').APIGatewayProxyHandler}
*/
export async function handler(event) {
console.log(`EVENT: ${JSON.stringify(event)}`)
if (!secretsLoaded && (await loadSecrets())) {
secretsLoaded = true
}
return {
statusCode: 200,
body: JSON.stringify(`Hello ${process.env.NAME}`),
}
}
```
With tracing enabled, let's look at the response times in order of appearance:
- `/hello-secrets` third request
- `/hello-secrets` second request
- `/hello-secrets` first request
- `/hello` only request

After rewriting env vars with secrets uploaded to ParameterStore, in order of appearance (**NOTE**: `/hello-secrets` has SSM logic commented out, however the SSM client import remained):
- `/hello` second request
- `/hello-secrets` second request
- `/hello` first request
- `/hello-secrets` first request

By removing the SSM fetch we are able to reduce the response time by 600ms. After removing the SSM Client library import entirely, the response times are the same, reducing response time by an additional 600ms.

Although I did not test bundling the Lambda function to a single entrypoint (with a tool like [esbuild](https://esbuild.github.io/), which is built into the [Lambda CDK Module's `NodeFunction` class](https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-lambda-nodejs.NodejsFunction.html)), circumventing fetching secrets at runtime reduces response times by about 1.2s. Using the secrets loading strategy shown in the sample handler above we are only loading secrets on the first invocation to reduce response times for subsequent requests, however the bottleneck lies with the first request on cold boot.
With the use case of a bot where we must ACK a response within 3s, fetching secrets at runtime can become problematic.
### Describe the solution you'd like
- use SSM for project-level, environment specific secrets using the nomenclature `/amplify///secrets/*`
- all functions are opted in to secrets created in parameter store
- after a successful push, function configuration is updated via AWS SDK with secrets fetched from SSM
### Describe alternatives you've considered
Using hooks:
- **pre-push.js**
- load from project-level `.env`, environment-specific `.env.[env-name]` as project override
- create secrets in SSM from loaded env files
- **post-push.js**
- iterate over functions in project, update configuration for each adding all env vars from SSM by the base name (e.g. `MY_VAR`)
Without hooks:
Using a similar strategy above where we are loading secrets from `.env*` files we can create a lightweight `secrets` category that creates/updates a `_snapshot` secret of the file hash
```js
export async function generateFileHash() {
const projectInfo = await getProjectInfo()
const mainEnvFilePath = path.resolve('.env')
const envSpecificEnvFilePath = path.resolve(`.env.${projectInfo.envName}`)
const hashSum = crypto.createHash('sha256')
if (await exists(mainEnvFilePath)) {
hashSum.update(await fs.readFile(mainEnvFilePath))
}
if (await exists(envSpecificEnvFilePath)) {
hashSum.update(await fs.readFile(envSpecificEnvFilePath))
}
const hex = hashSum.digest('hex')
return hex
}
```
And create the `_snapshot` parameter with the name `/amplify///secrets/_snapshot`
```ts
const snapshot = new ssm.StringParameter(this, 'SecretsSnapshotParameter', {
parameterName: SNAPSHOT_PARAMETER_NAME,
stringValue: HEX,
})
```
### Additional context
proof-of-concept https://github.com/josefaidt/amplify-secrets-using-hooks
**NOTE**: example does not delete parameters from SSM when they are removed from env dotfiles, and are ignored
### Is this something that you'd be interested in working on?
- [X] 👋 I may be able to implement this feature request
### Would this feature include a breaking change?
- [X] ⚠️ This feature might incur a breaking change
Contributor guide
Research direction
Start by tracing the CLI's function-secret handling and push flow, then compare the proposed pre-push.js and post-push.js hook strategy with the AWS SDK configuration update path. Review the proof of concept and the SSM naming and _snapshot examples. Done means secrets are synchronized to Lambda environment variables after a successful push, including the documented removal behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, node.js, typescript
- Domain
- backend, cli, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100