influxdata / influxdata/influxdb
Add ability to do historical backfill
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
As mentioned in #24745 the server will now only allow writing to 100 open segments at any given time. This was done to simplify the design of how ingested data is segmented as written up in #24706. Depending on the segment duration this could be only 100 minutes to 800 hours back in time. We'll need to support historical backfill.
We have a few options for how to do this. One would be through a new API endpoint that decides how to write data into the existing segment range. Or we could have the regular write API direct any overflow data (i.e. data with timestamps outside the currently 100 open segments) into some special segment.
In either case, this additional data would end up having to be combined at query time with any segments it overlaps with, which could be a big performance hit. This likely won't be solved without the compactor, which is outside the scope of this open source project.
This issue is open for tracking/discussion. We may ultimately decide that streaming historical backfill isn't in the scope of this project.
Contributor guide
Research direction
Read issues #24745 and #24706 first to understand the 100-open-segment constraint and the segmentation design. This issue names no files, tests, or entry points; work is not ready until the project decides whether historical backfill is in scope and selects a concrete API or write-path design.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100