influxdata / influxdata/influxdata-docker

Runaway memory causes container to crash (Unexpected end of JSON input via node-influx)

Open
#313 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
364
Forks
255
Avg merge
3h 18m
Merged PRs (30d)
7

Description

We're running the standard base image of influx, and when we run queries that have large string fields, it often crashes the influx container. We can watch the memory utilization increase to 80+ percent very quickly after even a single query (1GB allocated to the container).

Here's an example of the query:

 influx  2019-03-06T00:57:01.634595Z    info    Executing query {"log_id": "0E07sns0000", "service": "query", "query": "SELECT time, type, uuid, last(data) AS data FROM tend.\"52w\".Events WHERE type = 'PROGRAM' AND time < 1551373464022000000"}
 influx  /start.sh: line 10:   137 Killed                  influxd -config /influxdb.conf

The error as seen from node-influx running in a separate container:
ERROR: Unexpected end of JSON input

Here's our compose file:

influx:
    restart: always
    # can't mount the fs without being privileged
    privileged: true
    build: ./influx
    network_mode: 'host'
    volumes:
      - 'influx-data:/data'

Here's the dockerfile:

FROM resin/%%RESIN_MACHINE_NAME%%-golang:1.9-slim

# Some setup for apt-get
RUN apt-get update \
    && apt-get install apt-transport-https \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Setup to obtain the packages through apt-get
RUN curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
RUN echo "deb https://repos.influxdata.com/debian jessie stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

# InfluxDB
RUN apt-get update \
    && apt-get install influxdb chronograf --fix-missing \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
    
COPY influxdb.conf /

# enable container init system.
ENV INITSYSTEM on

COPY start.sh /

CMD ["bash", "/start.sh"]

We haven't changed the default configuration file - only modified where we store the data.

I've seen references to TSM1 and memory leaks, but we're using inmem.

Are there any ways to control the memory utilization of influx other than limiting concurrency and rejecting queries? Any other ideas are appreciated.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Dockerfile, influxdb.conf, and start.sh, then reproduce the reported query against the standard image while monitoring the container's memory. A useful resolution would identify a reproducible cause and a documented configuration or image change that prevents the container from being killed.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, shell
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.