influxdata / influxdata/influxdata-docker
Runaway memory causes container to crash (Unexpected end of JSON input via node-influx)
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 364
- Forks
- 255
- Avg merge
- 3h 18m
- Merged PRs (30d)
- 7
Description
We're running the standard base image of influx, and when we run queries that have large string fields, it often crashes the influx container. We can watch the memory utilization increase to 80+ percent very quickly after even a single query (1GB allocated to the container).
Here's an example of the query:
influx 2019-03-06T00:57:01.634595Z info Executing query {"log_id": "0E07sns0000", "service": "query", "query": "SELECT time, type, uuid, last(data) AS data FROM tend.\"52w\".Events WHERE type = 'PROGRAM' AND time < 1551373464022000000"}
influx /start.sh: line 10: 137 Killed influxd -config /influxdb.conf
The error as seen from node-influx running in a separate container:
ERROR: Unexpected end of JSON input
Here's our compose file:
influx:
restart: always
# can't mount the fs without being privileged
privileged: true
build: ./influx
network_mode: 'host'
volumes:
- 'influx-data:/data'
Here's the dockerfile:
FROM resin/%%RESIN_MACHINE_NAME%%-golang:1.9-slim
# Some setup for apt-get
RUN apt-get update \
&& apt-get install apt-transport-https \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Setup to obtain the packages through apt-get
RUN curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
RUN echo "deb https://repos.influxdata.com/debian jessie stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
# InfluxDB
RUN apt-get update \
&& apt-get install influxdb chronograf --fix-missing \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
COPY influxdb.conf /
# enable container init system.
ENV INITSYSTEM on
COPY start.sh /
CMD ["bash", "/start.sh"]
We haven't changed the default configuration file - only modified where we store the data.
I've seen references to TSM1 and memory leaks, but we're using inmem.
Are there any ways to control the memory utilization of influx other than limiting concurrency and rejecting queries? Any other ideas are appreciated.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Dockerfile, influxdb.conf, and start.sh, then reproduce the reported query against the standard image while monitoring the container's memory. A useful resolution would identify a reproducible cause and a documented configuration or image change that prevents the container from being killed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, shell
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100