KCIDB exceeds BigQuery interactive query limits
- Dominant language
- Python
- Stars
- 26
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
BigQuery has a limit of 100 concurrent "interactive" queries per Google Cloud project. We've been exceeding that, now that syzbot is sending its results to the playground instance. This was especially seen during notification generation, so as a temporary measure I disabled retrials there, to give up notification generation after first failure and thus avoid continuing to spam the database with queries.
The proper fix for this would be adding a less-scalable, but more-interactive database as our notification and dashboard data source, with recent data only, while keeping pushing all data to BigQuery.
Other temporary measures could be minimizing the number of queries in the notification generation. E.g. by pushing bigger, bundled, and not smaller, original messages there. We could also review the number of queries we're doing there.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by tracing notification generation and dashboard query paths, then review how retries and query bundling work. Done means choosing and implementing a sustainable data-source or query-volume approach that avoids BigQuery interactive-query limits while retaining the full data flow to BigQuery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100