influxdata / influxdata/influxdb
Kapacitor batch on large dataset over http fails
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
A kapacitor batch reading from one influxdb retention policy over http and writing to another will fail with the message :
task=my_batch_task node=query1 err="failed to decode JSON: unexpected EOF"
If the query takes > 35s to begin returning results. If the query starts returning results before the 35s, it will complete successfully, regardless of how long it takes to send the results back to the client
Can also replicate against the http api with curl.
InfluxDB query timeout is set to 0s (i.e. do not timeout)
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. Populate a measurement with a large number of points e.g. 5 million points over a 5 minute window
2. Attempt to do an aggregate query against the measurement via http api e.g. "select mean(cpu_used) from autogen.my_measurement group by time(5m), host"
3. Confirm if the query takes >35s to start returning results that the message above is seen
__Expected behavior:__
InfluxDB is set to not timeout - query should run to completion.
__Actual behavior:__
Error returned after 35s "failed to decode JSON: unexpected EOF"
__Environment info:__
* System info: Run `uname -srm` and copy the output here
* InfluxDB version: Run `influxd version` and copy the output here
* Other relevant environment details: Container runtime, disk info, etc
InfluxDB Version 1.7.8
__Config:__
Copy any non-default config values here or attach the full config as a gist or file.
__Logs:__
kapacitor-metrics.2019-12-19.log:ts=2019-12-19T16:45:30.355Z lvl=error msg="error executing query" service=kapacitor task_master=main task=my_batch_task node=query1 err="failed to decode JSON: unexpected EOF"
influxdb.log : ts=2019-12-19T00:00:10.831682Z lvl=warn msg="Detected slow query: SELECT max(value) AS value FROM prometheus.autogen.my_measurement WHERE time >= '2019-12-18T23:50:00Z' AND time < '2019-12-18T23:55:00Z' GROUP BY time(5m, 0s), host (qid: 26167, database: , threshold: 10s)" log_id=0J_CRlZl000 service=query
_
_Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.
```sh
# Commands should be run when the bug is actively happening.
# Note: This command will run for at least 30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=true"
curl -o vars.txt "http://localhost:8086/debug/vars"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz`, `vars.txt`, and `iostat.txt` output files.
```
Contributor guide
Research direction
Reproduce the failure using the HTTP API with curl and the large aggregate query described in the issue, then compare behavior when results begin before versus after 35 seconds. Trace the InfluxDB HTTP query path and the Kapacitor batch request handling; done means the query completes without the JSON unexpected-EOF error when query timeout is 0s.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100