influxdata / influxdata/influxdb

Migration from AWS cloud to Azure cloud subscription fails with data loss

Open
#25,270 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Steps to reproduce:

  1. Follow the migration guide as per the official documentation.
  2. Compare the size of the tables in the source and target databases after migration.
  3. Observe the data loss (30-40% missing).

Expected behaviour:

I have been following the official migration guide for InfluxDB v2 as outlined (https://docs.influxdata.com/influxdb/v2/write-data/migrate-data/migrate-cloud-to-oss/). I attempted the migration twice, using both methods provided in the documentation.

Although I did not encounter any migration errors during the process, I observed that the data was not fully migrated. Specifically, when comparing the table sizes between the source and target databases, I found that 30-40% of the data was missing in the target. There were no duplicates present in the source table, so that does not explain the discrepancy.

Actual behaviour:

Huge data losses in both cases, no errors in the logs

Environment info:

  • Source bucket in the AWS cloud
  • Destination bucket in the Azure cloud

Config:

  • start: 2023-12-01T00:00:00Z,
  • stop: 2024-08-31T00:00:00Z,
  • batchInterval: 1h,
  • option task = {
    name: "Migration",
    every: 30s,
    }

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the official InfluxDB v2 cloud-to-OSS migration guide and reproduce both documented migration methods using the reported AWS source, Azure destination, and date range. Compare source and target table sizes and inspect migration logs; done means the migration preserves the source data or clearly reports any failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, azure
Domain
cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.