influxdata / influxdata/influxdb

Missing tsm file in shard makes writes fail with (unclear) error message "[shard x] unexpected end of JSON input"

Open
#23,628 2 comments 5 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
1. Harddrive is full preventing influxdb successfully writing data
(log: error="write /var/lib/influxdb/engine/data/e7adcc11582b82f9/autogen/222/index/3/L1-00000001.tsi: no space left on device" path=/var/lib/influxdb/engine/data/e7adcc11582b82f9/autogen/222/index/3/L0-00000001.tsl)
2. You end up with corrupt data/shard
3. Restart influxdb after clean up of harddrive, there is now space to write to
4. There is now a shard without a .tsm/.tsl file
5. Try to write data to the time period of the above mentioned shard

__Expected behavior:__
The data is successfully written (the database will handle the missing .tsm file and do what is necessary to be able to write new data).

__Actual behavior:__
Receive error message
(python): HTTP response body: {"code":"internal error","message":"unexpected error writing points to database: [shard 222] unexpected end of JSON input"}
(log): Aug 16 13:08:05 COMPUTER influxd-systemd-start.sh[224631]: ts=2022-08-16T11:08:05.063037Z lvl=warn msg="Write failed creating shard" log_id=0cLaEUw0000 service=storage-engine service=write shard=222 error="opening shard previously failed with: [shard 222] unexpected end of JSON input"

__Environment info:__

* System info: Linux 5.10.0-10-amd64 x86_64
* InfluxDB version: InfluxDB v2.3.0+SNAPSHOT.090f681737 (git: 090f681737) build_date: 2022-06-16T19:33:50Z
* Other relevant environment details: N/A

__Config:__
Default

__Other info:__
Solved by deleting the shard folder in /engine/data/xxxx/autogen (in my case 222).

Contributor guide

Open the contributing guide

Research direction

Use the reproduction steps and the shard path under /engine/data/xxxx/autogen, focusing on the shard-opening and write path when the .tsm/.tsl file is missing. Confirm the failure shown in the logs, then verify that writing to the affected time period succeeds without requiring deletion of the shard folder.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.