influxdata / influxdata/influxdb

unexpected fault address resulted in 7 day and 2 hour data loss

Open
#24,018 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
¯\\\_(ツ)\_/¯

1. run influxdb for a few months and hope it happens to you?

note: i've had this happen before
i think it was always a loss of ~7 days, and always not the most recent 7 days
(just looking back, i found a loss from 2022-03-06 23:00:00 to 2022-03-14 02:00:00 on another bucket)
i think i had more than these two, maybe something got recovered automatically?

__Expected behavior:__
influxdb to either not crash, or crash but not lose data

__Actual behavior:__
influxdb, apparently spontaneously, decided to segfault
it lost 7 days and 2 hours of data in some measurements in 1 bucket
it lost this for a period approximately 4 days before the crash
(so going backwards in time, i have:
today's data up to the crash (2022-12-30 02:46:50)
data ~4 days before the crash (2022-12-26 01:00:00)
7 days and 2 hours missing (2022-12-18 23:36:20)
older data is still there)

__Environment info:__

* System info: Linux 6.1.0 x86_64
* NixOS unstable
* InfluxDB version: InfluxDB 2.5.1 (git: v2.5.1) build_date: 2022-12-30T20:33:07Z
* reported build_date is time of executing `influxd version`

__Config:__
apparently nothing non-stock, except a mistyped attempt to disable telemetry reporting

__Logs:__
[influxdb2-crash-trimmed.log](https://github.com/influxdata/influxdb/files/10326174/influxdb2-crash-trimmed.log)

Contributor guide

Open the contributing guide

Research direction

Begin with the attached influxdb2-crash-trimmed.log and the reported InfluxDB 2.5.1 environment on Linux 6.1.0 x86_64 and NixOS unstable. Determine what caused the segfault and the missing-data interval; done means the crash and data loss are explained and prevented.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, nixos, rust
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.