influxdata / influxdata/influxdb

Silent flux failure while using to(), "internal error not returned to client"

Open
#24,055 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

I am attempting to side load ~2GB worth of backed up data from a temporary bucket to an existing bucket in my InfluxDB instance. To do this I am running the following flux query:

```flux
from(bucket: "tmp")
|> range(start: -1y)
|> filter(fn: (r) => r._measurement !~ /event_logs/)
|> to(bucket: "dest")
|> map(fn: (r) => ({"_value": 1, "_time": r._time, "_measurement": r._measurement}))
|> group(columns: ["_measurement"])
|> count()
```

While running this query it sometimes appears the `influx query` CLI command hangs. During this I can see my instances memory usage and CPU usage drop (as if the query stopped, the influx instance usually maxes a single core and uses >8Gi of memory during this time) and I find this entry in the logs:

```
ts=2023-01-20T16:44:26.130913Z lvl=warn msg="internal error not returned to client" log_id=0fTme62G000
handler=error_logger error="context canceled"
```

Any ideas on how to make this restore process more reliable (and possibly faster)?

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. `influx backup` ~2GiB worth of data from a single bucket
2. `influx restore` into a temp bucket
3. Attempt to use Flux to sideload points into an existing bucket with data in it.

__Expected behavior:__
Sideloading to complete.

__Actual behavior:__
Indefinite hang.

__Environment info:__

* System info: `Linux 5.4.0-1098-azure x86_64` - Azure AKS, Standard_E4as_v4 node
* InfluxDB version: `InfluxDB v2.6.0 (git: 24a2b621ea) build_date: 2022-12-15T18:47:00Z`
* Other relevant environment details: N/A

__Config:__
```json
{
"assets-path": "",
"bolt-path": "/var/lib/influxdb2/influxd.bolt",
"e2e-testing": false,
"engine-path": "/var/lib/influxdb2",
"feature-flags": null,
"flux-log-enabled": true,
"hardening-enabled": false,
"http-bind-address": ":8086",
"http-idle-timeout": 86400000000000,
"http-read-header-timeout": 10000000000,
"http-read-timeout": 0,
"http-write-timeout": 0,
"influxql-max-select-buckets": 0,
"influxql-max-select-point": 0,
"influxql-max-select-series": 0,
"instance-id": "",
"log-level": "info",
"metrics-disabled": false,
"nats-max-payload-bytes": 0,
"nats-port": 4222,
"no-tasks": false,
"pprof-disabled": false,
"query-concurrency": 1024,
"query-initial-memory-bytes": 0,
"query-max-memory-bytes": 0,
"query-memory-bytes": 0,
"query-queue-size": 1024,
"reporting-disabled": false,
"secret-store": "bolt",
"session-length": 1440,
"session-renew-disabled": false,
"sqlite-path": "/var/lib/influxdb2/influxd.sqlite",
"storage-cache-max-memory-size": 1073741824,
"storage-cache-snapshot-memory-size": 26214400,
"storage-cache-snapshot-write-cold-duration": "10m0s",
"storage-compact-full-write-cold-duration": "4h0m0s",
"storage-compact-throughput-burst": 50331648,
"storage-max-concurrent-compactions": 0,
"storage-max-index-log-file-size": 1048576,
"storage-no-validate-field-size": false,
"storage-retention-check-interval": "30m0s",
"storage-series-file-max-concurrent-snapshot-compactions": 0,
"storage-series-id-set-cache-size": 0,
"storage-shard-precreator-advance-period": "30m0s",
"storage-shard-precreator-check-interval": "10m0s",
"storage-tsm-use-madv-willneed": false,
"storage-validate-keys": false,
"storage-wal-fsync-delay": "0s",
"storage-wal-max-concurrent-writes": 0,
"storage-wal-max-write-delay": 600000000000,
"storage-write-timeout": 10000000000,
"store": "disk",
"testing-always-allow-setup": false,
"tls-cert": "",
"tls-key": "",
"tls-min-version": "1.2",
"tls-strict-ciphers": false,
"tracing-type": "",
"ui-disabled": false,
"vault-addr": "",
"vault-cacert": "",
"vault-capath": "",
"vault-client-cert": "",
"vault-client-key": "",
"vault-client-timeout": 0,
"vault-max-retries": 0,
"vault-skip-verify": false,
"vault-tls-server-name": "",
"vault-token": ""
}
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied Flux query with the listed backup, restore, and influx query steps, then inspect the server logs for the "internal error not returned to client" and context-canceled behavior. Done means the sideload completes or the client receives a clear error instead of hanging.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.