influxdata / influxdata/influxdb

Corrupted buckets after server "power loss"

Open
#21,619 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/2.x area/storage
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Hello,

I'm using the latest InfluxDB 2 on a windows machine. I migrated last week from V1.8 which has been running flawlessly for the last 3 years.

This morning this windows server crashed as it sometimes do (it is a virtual machine, so it is not a physical hardware problem).
From this time Influxdb stopped writing on most buckets.

My problem is very similar to this one : https://github.com/influxdata/influxdb/issues/9949

I can see thousands of these two messages :
"write_errors"
[shard 1997] invalid character '\x00' looking for beginning of value
[shard 1993] invalid character '\x00' looking for beginning of value

I don't know how to repair this. I found the two folders 1997 and 1993, but I don't know how to handle this problem.

You can find attached the file that I found in the 1997 folder.
[fields.idx.txt](https://github.com/influxdata/influxdb/files/6605478/fields.idx.txt)

I tried to see if I could repair some tsm files like on the other ticket, but I have 273 tsm files.
Thank you by advance for your help.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the reported errors and attached fields.idx.txt from shard folders 1997 and 1993 with the prior issue #9949. Review how the 273 TSM files are handled after the Windows server power loss. Done is not defined in the report because it does not specify a supported repair procedure or an expected recovery result.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.