cube-js / cube-js/cube

Exponential backoff on errors

Open
#7,390 1 comment 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

**Is your feature request related to a problem? Please describe.**
Older version (v0.30) deployed to an environment that was upgraded from Athena V2 to V3. V3 includes a precision on timestamps `timestamp(3)` when describing the data type on columns from the information schema. A fix was deployed for this creating errors https://github.com/cube-js/cube/pull/6324.

As a consequence of this error, pre-aggregations were being built and then deleted, creating a tremendous amount of activity in memory, on CPU, and on the FS. The volume of attempted (and failing work) remained steady until our monitoring systems alerted us to the anomaly.

**Describe the solution you'd like**
Some kind of exponential backoff or throttling in place for repeating identical queries.

**Describe alternatives you've considered**
By upgrading the deployed versions we're now compatible with the new Athena version, so the problem creating the need for an exponential back off is resolved.

**Additional context**
The underlying problem of Athena compatibility has been fixed, but this incident highlighted an opportunity to improve Cube's ability to mitigate risks by adding throttling or backoff on repeated failure. I can see two current issues raised for search term backoff: https://github.com/cube-js/cube/issues/2506 and https://github.com/cube-js/cube/issues/3904. In both cases, addressing the bug would reduce the need for backoff - but backoff is a nice insurance policy against unknown bugs or incompatibilities generating a lot of resource use.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.