brigadecore / brigadecore/brigade
Need to impose size/cardinality limits on more fields
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 244
- PR merge metrics
- No merged PRs in 30d
Description
Some fields in projects , events, and jobs have unbounded length (or in the case of maps/arrays, cardinality).
We should impose some reasonable (TBD) limits-- or at the very least return a helpful error messages when we bump into hard limits on the size of underlying k8s resources.
Rationale:
Workers discover relevant event details, which include event metadata (source, type, qualifiers, labels, etc.) embedded scripts / config (if applicable), project name, project secrets (if applicable), event payload (if applicable), and a token for communicating with the Brigade API server from a chunk of JSON that is written to a k8s `Secret` and mounted to the worker's file system.
Jobs also each get their own secret that contains any/all environment variables used by the job.
`Secret`s have a hard size limit of 1MiB.
Some fields whose size we might consider limiting:
* `workerTemplate.defaultConfigFiles` on a project definition. Enforce a max number of files, each with a max size? Or enforce a cumulative max size?
* `qualifiers` on an event-- enforce a max number of qualifiers?
* `labels` on an event-- enforce a max number of labels?
* `payload` on an event
* env vars on a job-- enforce a max number of env vars, each with a max size? Or enforce a cumulative max size?
I'm sure there are many more.
I think this could get complicated quickly. Figuring out reasonable limits on map/array cardinality or limits on individual field lengths so they add up to a certain max size sounds tricky. And if we work it all out and don't leave room for expansion (new fields) someday? Then what?
There are also a lot of permutations here. Should someone be able to embed a larger script if their payloads are always small? Larger payloads if their scripts are small?
So, instead of putting hard limits on the cardinality of different data structures or the length of individual fields, maybe we should consider simply returning appropriate error codes and detailed messages when Brigade itself bumps into the limits imposed by k8s?
Contributor guide
Assessment
This issue has not been assessed yet.