HCL2: Fetch data from consul, vault, etc
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
Consider adding job spec HCL functions for fetching data from vault/consul and other sources. Packer categorizes these as [Contextual Functions](https://www.packer.io/docs/from-1.5/functions/contextual/).
## Design Considerations
Nomad currently support fetching data from Vault and Consul via [Consul Template](https://www.nomadproject.io/docs/job-specification/template). When adding contextual HCL functions, they should co-exist nicely with that support - operators must have clear expectations of the behavior of each and don't get surprised when they switch between them.
We have two open questions for supporting contextual functions.
### Evaluation Context
Contextual variables present a design challenge: How should they be evaluated?
Current HCL2 functions are evaluated statically by the CLI on the job submitter host. By implementing `vault`/`consul` functions as a normal HCL function, the CLI must be configured to access the production Vault/Consul. The CLI must reach the production Vault/Consul cluster, and the operator must have direct read access to production secrets. Such production access may not be ideal.
An alternative is to have Nomad servers fetch Vault/Consul values on behalf of the job. This is consistent with the current Vault/Consul integration with consul templates, and eases the deployment flow.
### Leases and Refresh
The documentation must set clear expectations on refresh behavior. Current, Consul Template integration have watch semantics, so a task may be notified when a vault/consul value changes. Also, Consul Template integration now refreshes the lease of the Vault secret.
As HCL functions are evaluated statically, a simple contextual function will only be evaluated once at submission time. Operators shouldn't expect the jobs to be updated when the vault/consul source data is changed. Also, they should expect the jobs to potentially run past the lease expiry of the fetched secrets, potentially causing a service outage.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the job-specification template documentation and the Packer contextual-functions documentation linked in the issue, then compare their evaluation and refresh behavior. Done means a decided design for Vault and Consul functions, including whether the CLI or Nomad servers fetch values and clear expectations for leases, watches, and updates.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, distributed-systems, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100