influxdata / influxdata/influxdb

InfluxDB 2: HTTP timeouts showing up as write errors

Open
#26,957 0 comments 0 reactions 1 assignee Claimed by @devanbenz View on GitHub
area/2.x
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

I've been running InfluxDB 2 for several years and I can't immediately move to InfluxDB 3 -- too much reliance on Flux. I've built a meta-monitoring stack (Influx monitoring itself) and realised that the `storage_shard_write_err_sum` (before #25056 is fixed) field increments every time there is a write timeout.

__Steps to reproduce:__
List the minimal actions needed to reproduce the behaviour.

1. Push data to InfluxDB
2. Add latency while processing a batch change -- either I/O contention, memory pressure/swapping, stop the process etc, alternatively set a very strict HTTP timeout (1ms or something crazy)
3. Monitor the `storage_shard_write_err_sum` metric

__Expected behaviour:__
The `storage_shard_write_err` field should only increase for true errors that the client cannot do anything about (e.g out of disk space), and not for client retriable errors (like timeout)

To be clear: I'm not really concerned about the performance here; I'm happy for the timeouts to happen to signal some backpressure/partial write. It's just that this shouldn't trigger any monitoring alerts (c.f. 4xx vs 5xx HTTP errors)

__Actual behaviour:__
The metric increases every time a timeout happens.

__Environment info:__

* Please provide the command you used to build the project, including any `RUSTFLAGS`.
* System info: Ubuntu 24.04, running kernel `6.14.0-33-generic`
* Deployed on Kubernetes
* Other relevant environment details: 2x 6-wide RAIDZ2 on ZFS exposed to InfluxDB as a volume mount (via [zfs-localpv](https://github.com/openebs/zfs-localpv)). It's platter based and so when IOPS is hammered applications which are IO bound will take a performance hit.

__Config:__
This doesn't rely on config.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.