influxdata / influxdata/influxdb
Provide better support for containers - startup order is not resilient, config default path cannot be specified
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Prior to version 2, it was possible for the official `telegraf` container to be started independently of the `influxdb` container, as it did not rely on `influxd` to provide the configuration. In particular, the `telegraf` container could be started some time before the `influxdb` container and the system would remain stable and a successful connection be made, as recommended by the docker-compose startup-order [docs](https://docs.docker.com/compose/startup-order/):
> The problem of waiting for a database (for example) to be ready is really just a subset of a much larger problem of distributed systems. In production, your database could become unavailable or move hosts at any time. Your application needs to be resilient to these types of failures.
WIth 2.0.0-beta, it is now necessary for the `telegraf` service to wait long enough for the `influxdb` container to start and make the configuration file available over HTTP. If the `telegraf` container is started too early, it will fail to obtain the configuration and fail to start. There is no provision in the official `telegraf` container to wait or retry the connection in the event that the configuration is not available.
The effect of this is that managing the telegraf/influxdb stack with tools like `docker-compose` is now non-trivial, if official docker images are used. Instead, a user needs to construct their own containers with the appropriate wait-for/retry scripts wrapping `telegraf`.
I suggest a modification to the official `telegraf` container to wait for configuration to be available, by attempting to download the configuration URL and then backing off for a short period of time if it is not available. Perhaps after some number of attempts, the container might exit with a permanent error, however for true resilience it should probably continue to poll for the configuration file indefinitely.
The other non-container-friendly aspect I have discovered is how `influx setup` saves the initial configuration to `/root/.influxdbv2/configs`, which is typically not retained in the official `influxdb` container. I appreciate that this can be worked around with an appropriate volume export in `docker-compose.yml`. To make this more container-friendly, I suggest adding a command-line option to `influx setup` that allows the user to specify the destination path for this file, so that it can be more easily integrated with existing container volume mounts.
Contributor guide
Research direction
Start with the official telegraf container startup behavior and the `influx setup` command described in the issue. Check how configuration retrieval behaves before influxdb is ready and how the setup destination is currently chosen. Done means startup handles unavailable configuration as requested and `influx setup` supports specifying its configuration path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose
- Domain
- cli, databases, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100