Ingester queue is getting too deep
- Dominant language
- Python
- Stars
- 0
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Reported at https://discord.com/channels/1245820301053530313/1301896040349433957/1441194713188274176
Possible actionables
- Check where is the bottleneck (Python, PostgreSQL, or origins re-submitting data)
- If it is Python: Check if we could deploy multiple ingesters in parallel
- If it is PostgreSQL: Check if settings or indexes that would improve the consumption
- If it is origins: Check which origin is causing the bursts
- Rate-limit submissions
- Rewrite ingester in Rust, if we determine Python is the culprit
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Discord report and inspect the ingester path to determine whether Python, PostgreSQL, or origins re-submitting data is the bottleneck. Compare the possible mitigations listed in the issue—parallel ingesters, database settings or indexes, origin-specific investigation, rate limiting, or a Rust rewrite—and consider the issue complete when the cause is identified and a justified mitigation is implemented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python, rust
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100