influxdata / influxdata/influxdb

The API encounters a write timeout error while InfluxDB is persisting Parquet files.. Influxdb3 core

Open
#26,413 5 comments 0 reactions 0 assignees View on GitHub
v3
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Are the operations of persisting WAL files and persisting Parquet files asynchronous? I encountered a problem of write data timeout through the Golang API.

I've already tried many configuration options from the official documentation. none of them have solved my problem.

Problem description:
1. I set a timeout of 30s for my golang client.
2. Each time I call the WritePoints() interface to batch write about 100 pieces of data.
3. At the beginning, it will write normally for a period of time. The InfluxDB3 log is like this: 2025-05-12T05:19:03.724421Z INFO influxdb3_wal::object_store: flushing WAL buffer to object store host="u1" n_ops=1 min_timestamp_ns=170591598000000000 max_timestamp_ns=1735675440000000000 wal_file_number=11919
4. Everything is normal up to the above. But after writing for a period of time, InfluxDB3 starts to persist Parquet files. The log is like this: 2025-05-12T05:52:30.131025Z INFO influxdb3_write::write_buffer::queryable_buffer: Persisted parquet file: u1/dbs/NASDAQ_STOCK-4/interval_1_min-2/2024-12-11/00-00/0000119140.parquet
4. Once the operation of persisting Parquet files starts, my write operation seems to be blocked. The client starts to time out and it looks like it can't write at all. All write requests will time out. It can only write until the operation of persisting Parquet is completed (this may take several minutes).
Therefore, I would like to ask whether the operation of writing to the WAL file and the operation of persisting Parquet will block each other? Is there any way to allow me to write data normally when persisting Parquet?
My environment is as follows:
Start command: influxdb3 serve --node-id u1 --data-dir /mnt/p44pro/data --http-bind localhost:8086 --object-store file
Version:3.0.2
Others: I used a soft link to connect the WAL folder to a different hard drive from the Parquet files, and both hard drives are SSDs. Through the system status, it can also be seen that my CPU usage and disk I/O are not high throughout the process. So it should not be a hardware problem.

I'm a newbie who has just started using InfluxDB and I'm really in need of your professional help. Thanks

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the timeout with the reported `influxdb3 serve` command, file object store, 30-second Go client timeout, and batches of about 100 points. Compare WAL flushing and Parquet persistence in the InfluxDB3 logs; done means establishing whether Parquet persistence blocks writes and identifying a concrete, reproducible change or documentation outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rust
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.