influxdata / influxdata/influxdb
Silent flux failure while using to(), "internal error not returned to client"
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
I am attempting to side load ~2GB worth of backed up data from a temporary bucket to an existing bucket in my InfluxDB instance. To do this I am running the following flux query:
```flux
from(bucket: "tmp")
|> range(start: -1y)
|> filter(fn: (r) => r._measurement !~ /event_logs/)
|> to(bucket: "dest")
|> map(fn: (r) => ({"_value": 1, "_time": r._time, "_measurement": r._measurement}))
|> group(columns: ["_measurement"])
|> count()
```
While running this query it sometimes appears the `influx query` CLI command hangs. During this I can see my instances memory usage and CPU usage drop (as if the query stopped, the influx instance usually maxes a single core and uses >8Gi of memory during this time) and I find this entry in the logs:
```
ts=2023-01-20T16:44:26.130913Z lvl=warn msg="internal error not returned to client" log_id=0fTme62G000
handler=error_logger error="context canceled"
```
Any ideas on how to make this restore process more reliable (and possibly faster)?
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. `influx backup` ~2GiB worth of data from a single bucket
2. `influx restore` into a temp bucket
3. Attempt to use Flux to sideload points into an existing bucket with data in it.
__Expected behavior:__
Sideloading to complete.
__Actual behavior:__
Indefinite hang.
__Environment info:__
* System info: `Linux 5.4.0-1098-azure x86_64` - Azure AKS, Standard_E4as_v4 node
* InfluxDB version: `InfluxDB v2.6.0 (git: 24a2b621ea) build_date: 2022-12-15T18:47:00Z`
* Other relevant environment details: N/A
__Config:__
```json
{
"assets-path": "",
"bolt-path": "/var/lib/influxdb2/influxd.bolt",
"e2e-testing": false,
"engine-path": "/var/lib/influxdb2",
"feature-flags": null,
"flux-log-enabled": true,
"hardening-enabled": false,
"http-bind-address": ":8086",
"http-idle-timeout": 86400000000000,
"http-read-header-timeout": 10000000000,
"http-read-timeout": 0,
"http-write-timeout": 0,
"influxql-max-select-buckets": 0,
"influxql-max-select-point": 0,
"influxql-max-select-series": 0,
"instance-id": "",
"log-level": "info",
"metrics-disabled": false,
"nats-max-payload-bytes": 0,
"nats-port": 4222,
"no-tasks": false,
"pprof-disabled": false,
"query-concurrency": 1024,
"query-initial-memory-bytes": 0,
"query-max-memory-bytes": 0,
"query-memory-bytes": 0,
"query-queue-size": 1024,
"reporting-disabled": false,
"secret-store": "bolt",
"session-length": 1440,
"session-renew-disabled": false,
"sqlite-path": "/var/lib/influxdb2/influxd.sqlite",
"storage-cache-max-memory-size": 1073741824,
"storage-cache-snapshot-memory-size": 26214400,
"storage-cache-snapshot-write-cold-duration": "10m0s",
"storage-compact-full-write-cold-duration": "4h0m0s",
"storage-compact-throughput-burst": 50331648,
"storage-max-concurrent-compactions": 0,
"storage-max-index-log-file-size": 1048576,
"storage-no-validate-field-size": false,
"storage-retention-check-interval": "30m0s",
"storage-series-file-max-concurrent-snapshot-compactions": 0,
"storage-series-id-set-cache-size": 0,
"storage-shard-precreator-advance-period": "30m0s",
"storage-shard-precreator-check-interval": "10m0s",
"storage-tsm-use-madv-willneed": false,
"storage-validate-keys": false,
"storage-wal-fsync-delay": "0s",
"storage-wal-max-concurrent-writes": 0,
"storage-wal-max-write-delay": 600000000000,
"storage-write-timeout": 10000000000,
"store": "disk",
"testing-always-allow-setup": false,
"tls-cert": "",
"tls-key": "",
"tls-min-version": "1.2",
"tls-strict-ciphers": false,
"tracing-type": "",
"ui-disabled": false,
"vault-addr": "",
"vault-cacert": "",
"vault-capath": "",
"vault-client-cert": "",
"vault-client-key": "",
"vault-client-timeout": 0,
"vault-max-retries": 0,
"vault-skip-verify": false,
"vault-tls-server-name": "",
"vault-token": ""
}
```
Contributor guide
Research direction
Start by reproducing the supplied Flux query with the listed backup, restore, and influx query steps, then inspect the server logs for the "internal error not returned to client" and context-canceled behavior. Done means the sideload completes or the client receives a clear error instead of hanging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100