aws-amplify / aws-amplify/amplify-hosting
Hosting Secrets (SSM) break on any branch name containing #/hash with a misleading generic error
- Dominant language
- Dockerfile
- Stars
- 481
- Forks
- 123
- PR merge metrics
- No merged PRs in 30d
Description
### Before opening, please confirm:
- [x] I have checked to see if my question is addressed in the [FAQ](https://github.com/aws-amplify/amplify-hosting/blob/master/FAQ.md).
- [x] I have [searched for duplicate or closed issues](https://github.com/aws-amplify/amplify-hosting/issues?q=is%3Aissue+).
- [x] I have removed any sensitive information from my code snippets and submission.
### Amplify Hosting feature
Deployments
### Is your feature request related to a problem? Please describe:
Amplify Hosting "Secrets" feature silently and unconditionally fails on any branch whose name contains `#` - masked by a useless generic warning that sends users chasing a permissions problem that doesn't exist
## Summary
Amplify Hosting's per-app Secrets feature (Console > Hosting > Secrets, backed by SSM Parameter Store) is completely broken for any branch name containing a `#` character. The SSM lookup path is built by concatenating the **raw, unsanitized git branch name** straight into the parameter path, e.g.:
```
/amplify//#74--Deployment-Test/
```
`#` is not a legal character in an SSM parameter name (SSM only allows `a-zA-Z0-9_.-` and `/`). The SSM call fails on every single build on that branch, with no exceptions, no matter how the secret is scoped (branch-specific or "all branches").
Instead of surfacing that failure honestly, Amplify swallows it and prints a content-free, actively misleading warning:
```
[WARNING]: !Failed to set up process.env.secrets
```
That line gives zero indication that the real cause is a path-validation failure on the branch name. It reads exactly like an IAM/permissions problem, which is where any engineer debugging this will spend their time first, because that's the only lead the log gives them.
## Why this matters
`#NN--Title` is a completely ordinary, 100% valid git branch name. It's not exotic - it's a common convention for linking branches to issue trackers (we use it for GitHub issue autolinking). Amplify Hosting has no problem building, deploying, or generating a preview URL for such a branch; the build succeeds end to end. Only the Secrets subsystem breaks, silently, because one specific code path never sanitizes or validates the branch name before using it as a structured storage key. That's a basic input-validation bug sitting inside a feature whose entire job is handling credentials.
The practical effect: any team that names feature/preview branches with a `#` in them cannot use Secrets at all, cannot get a real error telling them why, and has to reverse-engineer the SSM path construction themselves to find out (which is what we did). There is no console warning, no documentation callout, and no validation at branch-creation or secret-creation time that would catch this before a build burns through it.
## Steps to Reproduce
1. Create an Amplify Hosting app connected to a GitHub repo.
2. Create/push a branch with `#` in its name, e.g. `#74--Deployment-Test`, and connect it in Amplify.
3. Add a secret in Console > Hosting > Secrets, scoped either to this branch specifically or to "All branches".
4. Trigger a build on that branch.
5. Observe the build log.
### Describe how you'd like this feature to work
## Expected Behavior
One of:
- The secret is retrieved and injected into `process.env` as documented, **or**
- If the branch name is genuinely incompatible with how Secrets are stored, Amplify rejects the branch name (or the secret configuration) up front with a clear, actionable error identifying the illegal character(s) - not a silent per-build failure discovered by trial and error.
## Actual Behavior
- The SSM parameter lookup path is built directly from the unescaped branch name and is therefore invalid.
- The call fails on **every** build, unconditionally, regardless of secret scope.
- The only signal in the build log is a generic, boilerplate warning that says nothing about path validation, `#`, or SSM at all:
```
[WARNING]: !Failed to set up process.env.secrets
```
- This is indistinguishable in the log from an actual IAM permissions failure, which is a materially different problem with a materially different fix. Anyone triaging this from the log alone will misdiagnose it.
## Root Cause (as far as we can tell from black-box testing)
The Secrets bootstrap step constructs an SSM Parameter Store path of the form `/amplify///` and passes the raw branch name straight through with no encoding, sanitization, or validation against SSM's allowed parameter-name character set (`a-zA-Z0-9_.-` and `/`). Since `#` is outside that set, the SSM API call fails, and the failure is caught and downgraded to a generic warning rather than surfaced with any detail.
## Impact
- Silent, total loss of a documented feature (Secrets) for an entire class of otherwise-valid, otherwise-fully-supported branch names.
- Misleading diagnostics actively point debugging effort in the wrong direction (permissions, not validation).
- No workaround exists within the feature itself - the only fix is to stop using Secrets entirely and fall back to plain App-level Environment Variables (which have their own tradeoff: no "sensitive" masking at rest the way Secrets provides).
- This is the kind of bug that costs every team who hits it real engineering hours, for a root cause that a single `parameterName.match(/^[a-zA-Z0-9_.\-\/]+$/)` check at secret-save time or branch-connect time would have caught instantly.
## Suggested Fix
Pick one, ideally both:
1. **Validate early, fail loud.** When a branch is connected or a secret is saved, validate that the resulting SSM parameter path is legal. If not, reject with a specific error naming the offending character(s) and the branch name involved - not a generic runtime warning three steps removed from the cause.
2. **Sanitize the path.** Percent-encode or otherwise transform illegal characters when building the SSM path, so branch names that are valid git/GitHub branch names but invalid SSM path segments don't silently break a feature that has nothing to do with SSM's naming rules from the user's perspective.
At minimum, fix the warning message. `!Failed to set up process.env.secrets` with no cause, no branch name, no parameter path, and no distinction between "no permissions" and "invalid path" is not an acceptable error message for a credentials feature failing on every build.
## Our "Workaround"
We've **stopped** using Hosting > Secrets
Contributor guide
Research direction
Start from the Hosting Secrets build-time SSM lookup and the warning that reports “Failed to set up process.env.secrets”; trace how the raw branch name becomes a parameter path. Confirm the failure for a branch containing `#`, then make the behavior either support that branch or reject it with an actionable validation error and a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100