Exponential backoff on errors
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
**Is your feature request related to a problem? Please describe.**
Older version (v0.30) deployed to an environment that was upgraded from Athena V2 to V3. V3 includes a precision on timestamps `timestamp(3)` when describing the data type on columns from the information schema. A fix was deployed for this creating errors https://github.com/cube-js/cube/pull/6324.
As a consequence of this error, pre-aggregations were being built and then deleted, creating a tremendous amount of activity in memory, on CPU, and on the FS. The volume of attempted (and failing work) remained steady until our monitoring systems alerted us to the anomaly.
**Describe the solution you'd like**
Some kind of exponential backoff or throttling in place for repeating identical queries.
**Describe alternatives you've considered**
By upgrading the deployed versions we're now compatible with the new Athena version, so the problem creating the need for an exponential back off is resolved.
**Additional context**
The underlying problem of Athena compatibility has been fixed, but this incident highlighted an opportunity to improve Cube's ability to mitigate risks by adding throttling or backoff on repeated failure. I can see two current issues raised for search term backoff: https://github.com/cube-js/cube/issues/2506 and https://github.com/cube-js/cube/issues/3904. In both cases, addressing the bug would reduce the need for backoff - but backoff is a nice insurance policy against unknown bugs or incompatibilities generating a lot of resource use.
Contributor guide
Assessment
This issue has not been assessed yet.