influxdata / influxdata/influxdb

[v3] influxdb3 core fails to start with corrupt catalog and corrupt wal after unclean shutdown (power outage)

Open
#26,970 23 comments 5 reactions 0 assignees View on GitHub
v3
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behaviour.

1. influxdb3 abnormal stop due to server power outage
2. server power on, restart influxdb3 and application

__Expected behaviour:__
influxdb3 automatically start and write buffer as normal

__Actual behaviour:__
The issue is very similar to https://github.com/influxdata/influxdb/issues/26549
The difference is that I found both catalog and wal corrupt

1. When catalog corrupt, influxdb3 core cannot start. After fix catalog corrupt by remove empty catalog, influxdb3 could start.
2. When there is wal corrupt, influxdb3 core start with warning information (as fix in https://github.com/influxdata/influxdb/pull/26556). When application write buffer to influxdb3 core, a "File exists" error reported and influxdb3 shutdown. Since I set influxdb3 to auto-restart, this situation is continuing, until I fix the wal corrupt by remove the empty wal files.

Currently, I add a check script to remove empty catalog and wal before start influxdb3 core as workaround.
I am not sure if other files (index or parquet) may have the same issue when influxdb abnormal shutdown?

__Environment info:__
Docker Image influxdb:3-core
I tried latest image (influxdb 3.6.0) and old one (influxdb 3.4.1), both have the same behaviour.
Logs is from influxdb 3.6.0.

__Config:__
default config

__Logs:__
----_when catalog corrupt_
```
2025-11-12T06:11:33.334385Z INFO influxdb3_lib::commands::serve: Creating shared query executor num_threads=8
2025-11-12T06:11:33.441610Z WARN executor: DedicatedExecutor dropped without calling shutdown()
2025-11-12T06:11:33.441703Z WARN executor: DedicatedExecutor dropped without waiting for worker termination
2025-11-12T06:11:33.443185Z WARN executor: DedicatedExecutor dropped without calling shutdown()
2025-11-12T06:11:33.443243Z WARN executor: DedicatedExecutor dropped without waiting for worker termination
2025-11-12T06:11:33.444338Z INFO influxdb3_cache::parquet_cache: cache request handler closed
Serve command failed: failed to initialize catalog: object store error: Unexpected(failed to load catalog

Caused by:
0: unexpected error: failed to deserialize catalog file

Caused by:
0: unexpected error: file must contain at least 10 bytes
1: file must contain at least 10 bytes
1: failed to deserialize catalog file
2: unexpected error: file must contain at least 10 bytes
3: file must contain at least 10 bytes)
```

----_when wal corrupt_
```
2025-11-12T06:35:13.771001Z WARN influxdb3_wal::object_store: Skipping corrupt WAL file error=WAL file too small: expected at least 12 bytes, but got 0 bytes path=test/wal/00000001784.wal
2025-11-12T06:35:13.771044Z WARN influxdb3_wal::object_store: Skipping corrupt WAL file error=WAL file too small: expected at least 12 bytes, but got 0 bytes path=test/wal/00000001785.wal
2025-11-12T06:35:13.771051Z WARN influxdb3_wal::object_store: Skipping corrupt WAL file error=WAL file too small: expected at least 12 bytes, but got 0 bytes path=test/wal/00000001786.wal
2025-11-12T06:35:13.771057Z WARN influxdb3_wal::object_store: Skipping corrupt WAL file error=WAL file too small: expected at least 12 bytes, but got 0 bytes path=test/wal/00000001787.wal
2025-11-12T06:35:13.771062Z INFO influxdb3_wal::object_store: completed replaying wal files time_taken=252.467447ms
2025-11-12T06:35:13.771149Z INFO influxdb3_lib::commands::serve: setting up background mem check for query buffer
2025-11-12T06:35:13.771182Z INFO influxdb3_lib::commands::serve: setting up telemetry store
2025-11-12T06:35:13.771346Z INFO influxdb3_write::deleter: Started catalog hard deleter task. delete_grace_period=86400s
2025-11-12T06:35:18.935983Z INFO influxdb3_lib::commands::serve: setting up server with authz disabled for paths paths_without_authz=[]
2025-11-12T06:35:18.936056Z INFO influxdb3_server: startup time: 5527ms address=0.0.0.0:8181
2025-11-12T06:35:25.772321Z INFO influxdb3_wal::object_store: flushing WAL buffer to object store host="test" n_ops=1 min_timestamp_ns=1762929130197890000 max_timestamp_ns=1762929152949256000 wal_file_number=1784
2025-11-12T06:35:25.779164Z ERROR influxdb3_wal::object_store: invoking shutdown after attempt to persist a WAL file that already exists on the object store path="/.influxdb3/test/wal/00000001784.wal" source=Os { code: 17, kind: AlreadyExists, message: "File exists" }
2025-11-12T06:35:25.779372Z INFO influxdb3_write::retention_period_handler: Retention period handler shutting down
2025-11-12T06:35:25.779354Z INFO influxdb3_catalog::catalog::versions::v2: updating node state to stopped in catalog node_id="test"
2025-11-12T06:35:25.779433Z INFO influxdb3_catalog::catalog::versions::v2::update: updating node state to Stopped in catalog node_id="test" process_uuid=44107b7d-43ba-48fd-abea-78831792a9bf
2025-11-12T06:35:25.779839Z INFO influxdb3_catalog::object_store::versions::v2: persisted next catalog sequence put_result=PutResult { e_tag: Some("20b9547-6435ff839f38b-106"), version: None } object_path=CatalogFilePath(Path { raw: "test/catalog/v2/logs/00000000000000007392.catalog" })
2025-11-12T06:35:25.780417Z INFO influxdb3_wal::object_store: flushing WAL buffer to object store host="test" n_ops=1 min_timestamp_ns=1762929152870351000 max_timestamp_ns=1762929153500535000 wal_file_number=1785
2025-11-12T06:35:25.780633Z INFO influxdb3_write::deleter: Shutdown signal received, exiting object deleter loop.
2025-11-12T06:35:25.780825Z ERROR influxdb3_wal::object_store: invoking shutdown after attempt to persist a WAL file that already exists on the object store path="/.influxdb3/test/wal/00000001785.wal" source=Os { code: 17, kind: AlreadyExists, message: "File exists" }
2025-11-12T06:35:25.781657Z ERROR influxdb3_server::http: Error while handling request error=write buffer error: error from wal: wal is shutdown and not accepting writes method=POST path="/api/v3/write_lp" content_length=Some("65579") database=test client_ip=127.0.0.1
2025-11-12T06:35:25.781834Z INFO influxdb3_server: All connections closed gracefully
2025-11-12T06:35:25.781884Z INFO influxdb3_lib::commands::serve: frontend shutdown completed
2025-11-12T06:35:25.781908Z INFO influxdb3_lib::commands::serve: backend shutdown completed
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the unclean-shutdown scenario with the influxdb:3-core Docker image and inspect the catalog initialization and WAL replay paths identified in the logs. Check behavior for empty catalog and WAL files, including a subsequent write through /api/v3/write_lp. Done means the service starts after recovery and continues accepting writes without shutdown or "File exists" errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, rust
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.