influxdata / influxdata/influxdb

Provide better support for containers - startup order is not resilient, config default path cannot be specified

Open
#17,813 3 comments 2 reactions 0 assignees View on GitHub
area/docker team/user-tools
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Prior to version 2, it was possible for the official `telegraf` container to be started independently of the `influxdb` container, as it did not rely on `influxd` to provide the configuration. In particular, the `telegraf` container could be started some time before the `influxdb` container and the system would remain stable and a successful connection be made, as recommended by the docker-compose startup-order [docs](https://docs.docker.com/compose/startup-order/):

> The problem of waiting for a database (for example) to be ready is really just a subset of a much larger problem of distributed systems. In production, your database could become unavailable or move hosts at any time. Your application needs to be resilient to these types of failures.

WIth 2.0.0-beta, it is now necessary for the `telegraf` service to wait long enough for the `influxdb` container to start and make the configuration file available over HTTP. If the `telegraf` container is started too early, it will fail to obtain the configuration and fail to start. There is no provision in the official `telegraf` container to wait or retry the connection in the event that the configuration is not available.

The effect of this is that managing the telegraf/influxdb stack with tools like `docker-compose` is now non-trivial, if official docker images are used. Instead, a user needs to construct their own containers with the appropriate wait-for/retry scripts wrapping `telegraf`.

I suggest a modification to the official `telegraf` container to wait for configuration to be available, by attempting to download the configuration URL and then backing off for a short period of time if it is not available. Perhaps after some number of attempts, the container might exit with a permanent error, however for true resilience it should probably continue to poll for the configuration file indefinitely.

The other non-container-friendly aspect I have discovered is how `influx setup` saves the initial configuration to `/root/.influxdbv2/configs`, which is typically not retained in the official `influxdb` container. I appreciate that this can be worked around with an appropriate volume export in `docker-compose.yml`. To make this more container-friendly, I suggest adding a command-line option to `influx setup` that allows the user to specify the destination path for this file, so that it can be more easily integrated with existing container volume mounts.

Contributor guide

Open the contributing guide

Research direction

Start with the official telegraf container startup behavior and the `influx setup` command described in the issue. Check how configuration retrieval behaves before influxdb is ready and how the setup destination is currently chosen. Done means startup handles unavailable configuration as requested and `influx setup` supports specifying its configuration path.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose
Domain
cli, databases, devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.