google / google/go-jsonnet

API for providing ExtVars and TLAs lazily

Open
#338 9 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
1.8k
Forks
263
PR merge metrics
No merged PRs in 30d

Description

Inspired by https://github.com/grafana/tanka/issues/101, I was thinking about the possibilities of making data loading more convenient in Jsonnet.

Currently ExtVars and TLAs must be prepared fully before Jsonnet evaluation starts. This is fine when passing some simple parameters, but less so when passing big objects (e.g. contents of a directory, secret databases, etc.).

It may make sense to create a new API, which allows lazily calculating/fetching parts of the extVars/TLAs. So for example directory contents could be passed as an object, with its entries as fields. Only when a user accesses one of the fields, a relevant entry will be accessed. If it's a file then its contents will be returned as a string and if it's a directory, a lazy object will be returned.

Importantly, this does not change anything about the language semantics - we are already dealing with lazy values and evaluating them is already a potentially failing operation. For example in case of passing directory contents, if there is an IO error, the calculation will be immediately aborted. From the point of view of Jsonnet code, it's just data.

I think it can be a pretty good fit with proper caching, logging, and ability to pass a different data source (Jsonnet code only cares about what data is passed and not at all about the source).

I'm slightly worried that this will encourage passing a lot of parameters to Jsonnet programs - instead of keeping inputs and logic in the repo and using simple static imports. That said, currently users are tempted to use `std.native` for similar purposes - which is less portable and further from Jsonnet ideals than this proposal.

The most legitimate concrete use case is secret management. I can also imagine it used for integration with existing configs and for loading large lookup tables (perhaps with a static path `importdir` instead of TLAs or ExtVars).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.