influxdata / influxdata/influxdb

InfluxDB loses all data when moving from `latest` tag to `2.1`

Open
#23,194 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Hi, I deployed an InfluxDB instance with the `latest` tag on 2nd March 2022. Some days later (11th March) I reconsidered using `latest` and went with the tag `2.1`. In my understanding of SemVer, this should have included `2.1.1`. Then I ran `docker-compose pull` and `docker-compose up -d`. At this point, I was expecting nothing to be really changed, as `latest` should (as I thought) be equal with `2.1.1` (last release). To my surprise, I was greeted with the onboarding screen on the Web Interface, which asked me to set InfluxDB up. At that point, it seems like the user storage of InfluxDB was overwritten, and I was not able to get my data back, as it was partially overwritten by the onboarding.

### What I would have expected
I would have expected InfluxDB to be able to use the data, generated by `latest`, inside the last tagged release `2.1(.1)`.
As a fallback, I would have expected InfluxDB to warn me that there is existing data, that it can't read, and quit - not to corrupt the whole database.

I also expect a "downgrade" from `latest` to the last official release to work without errors, if not, changing from `latest` , to any other version, would not be possible.

### Possible solution

InfluxDB could check if there is existing data, and if there is, check if it can read it, before making any writes. Then it could validate the integrity of the data and only continue if the checks pass. If the data cannot be read, it could fail with an error message, explaining that it detected data that it cannot use. This could be a failsafe for any version change (up-/downgrade), without the possibility of losing data. The onboarding process should only start, if it's not overwriting existing data.

### Main Problem

Changing the tag from `latest` to the last release number, is something a user could understand as "pinning the current image version", to disable automatic upgrades on `docker pull`. In my case, I lost ~30 Million entries in the database and all configured dashboards / alerts / users / tokens. This is not toooo bad, as they were mostly system metrics over a week, but it's reasonable to say, that other users could get into way worse problems with such a little change to the docker-compose file.

I hope this helps! Let me know if you have questions.

> Since I loaded a full-server-backup of that container, I was not able to retrieve any logs.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the version change with docker-compose, moving an instance from the latest tag to 2.1, and observe the onboarding process. Trace how existing data is detected before onboarding writes occur. Done means incompatible existing data is not overwritten and the service reports a clear failure instead.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose
Domain
databases, devops
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.