influxdata / influxdata/influxdb

Kapacitor batch on large dataset over http fails

Open
#16,292 3 comments 0 reactions 0 assignees View on GitHub
1.x
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

A kapacitor batch reading from one influxdb retention policy over http and writing to another will fail with the message :
task=my_batch_task node=query1 err="failed to decode JSON: unexpected EOF"
If the query takes > 35s to begin returning results. If the query starts returning results before the 35s, it will complete successfully, regardless of how long it takes to send the results back to the client

Can also replicate against the http api with curl.

InfluxDB query timeout is set to 0s (i.e. do not timeout)

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. Populate a measurement with a large number of points e.g. 5 million points over a 5 minute window
2. Attempt to do an aggregate query against the measurement via http api e.g. "select mean(cpu_used) from autogen.my_measurement group by time(5m), host"
3. Confirm if the query takes >35s to start returning results that the message above is seen

__Expected behavior:__
InfluxDB is set to not timeout - query should run to completion.

__Actual behavior:__
Error returned after 35s "failed to decode JSON: unexpected EOF"

__Environment info:__

* System info: Run `uname -srm` and copy the output here
* InfluxDB version: Run `influxd version` and copy the output here
* Other relevant environment details: Container runtime, disk info, etc

InfluxDB Version 1.7.8

__Config:__
Copy any non-default config values here or attach the full config as a gist or file.

__Logs:__

kapacitor-metrics.2019-12-19.log:ts=2019-12-19T16:45:30.355Z lvl=error msg="error executing query" service=kapacitor task_master=main task=my_batch_task node=query1 err="failed to decode JSON: unexpected EOF"

influxdb.log : ts=2019-12-19T00:00:10.831682Z lvl=warn msg="Detected slow query: SELECT max(value) AS value FROM prometheus.autogen.my_measurement WHERE time >= '2019-12-18T23:50:00Z' AND time < '2019-12-18T23:55:00Z' GROUP BY time(5m, 0s), host (qid: 26167, database: , threshold: 10s)" log_id=0J_CRlZl000 service=query
_
_Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.

```sh
# Commands should be run when the bug is actively happening.
# Note: This command will run for at least 30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=true"
curl -o vars.txt "http://localhost:8086/debug/vars"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz`, `vars.txt`, and `iostat.txt` output files.
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure using the HTTP API with curl and the large aggregate query described in the issue, then compare behavior when results begin before versus after 35 seconds. Trace the InfluxDB HTTP query path and the Kapacitor batch request handling; done means the query completes without the JSON unexpected-EOF error when query timeout is 0s.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.