kernelci / kernelci/kcidb

KCIDB exceeds BigQuery interactive query limits

Open
#130 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
26
Forks
40
PR merge metrics
No merged PRs in 30d

Description

BigQuery has a limit of 100 concurrent "interactive" queries per Google Cloud project. We've been exceeding that, now that syzbot is sending its results to the playground instance. This was especially seen during notification generation, so as a temporary measure I disabled retrials there, to give up notification generation after first failure and thus avoid continuing to spam the database with queries.

The proper fix for this would be adding a less-scalable, but more-interactive database as our notification and dashboard data source, with recent data only, while keeping pushing all data to BigQuery.

Other temporary measures could be minimizing the number of queries in the notification generation. E.g. by pushing bigger, bundled, and not smaller, original messages there. We could also review the number of queries we're doing there.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by tracing notification generation and dashboard query paths, then review how retries and query bundling work. Done means choosing and implementing a sustainable data-source or query-volume approach that avoids BigQuery interactive-query limits while retaining the full data flow to BigQuery.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.