influxdata / influxdata/influxdb

Add ability to do historical backfill

Open
#24,752 1 comment 1 reaction 0 assignees View on GitHub
v3
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

As mentioned in #24745 the server will now only allow writing to 100 open segments at any given time. This was done to simplify the design of how ingested data is segmented as written up in #24706. Depending on the segment duration this could be only 100 minutes to 800 hours back in time. We'll need to support historical backfill.

We have a few options for how to do this. One would be through a new API endpoint that decides how to write data into the existing segment range. Or we could have the regular write API direct any overflow data (i.e. data with timestamps outside the currently 100 open segments) into some special segment.

In either case, this additional data would end up having to be combined at query time with any segments it overlaps with, which could be a big performance hit. This likely won't be solved without the compactor, which is outside the scope of this open source project.

This issue is open for tracking/discussion. We may ultimately decide that streaming historical backfill isn't in the scope of this project.

Contributor guide

Open the contributing guide

Research direction

Read issues #24745 and #24706 first to understand the 100-open-segment constraint and the segmentation design. This issue names no files, tests, or entry points; work is not ready until the project decides whether historical backfill is in scope and selects a concrete API or write-path design.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.