kernelci / kernelci/kcidb-ng

Ingester queue is getting too deep

Open
#56 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
6
PR merge metrics
No merged PRs in 30d

Description

Reported at https://discord.com/channels/1245820301053530313/1301896040349433957/1441194713188274176

Image

Possible actionables

- Check where is the bottleneck (Python, PostgreSQL, or origins re-submitting data)
- If it is Python: Check if we could deploy multiple ingesters in parallel
- If it is PostgreSQL: Check if settings or indexes that would improve the consumption
- If it is origins: Check which origin is causing the bursts
- Rate-limit submissions
- Rewrite ingester in Rust, if we determine Python is the culprit

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Discord report and inspect the ingester path to determine whether Python, PostgreSQL, or origins re-submitting data is the bottleneck. Compare the possible mitigations listed in the issue—parallel ingesters, database settings or indexes, origin-specific investigation, rate limiting, or a Rust rewrite—and consider the issue complete when the cause is identified and a justified mitigation is implemented.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python, rust
Domain
backend, databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.