influxdata / influxdata/influxdb
Migration from AWS cloud to Azure cloud subscription fails with data loss
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Steps to reproduce:
- Follow the migration guide as per the official documentation.
- Compare the size of the tables in the source and target databases after migration.
- Observe the data loss (30-40% missing).
Expected behaviour:
I have been following the official migration guide for InfluxDB v2 as outlined (https://docs.influxdata.com/influxdb/v2/write-data/migrate-data/migrate-cloud-to-oss/). I attempted the migration twice, using both methods provided in the documentation.
Although I did not encounter any migration errors during the process, I observed that the data was not fully migrated. Specifically, when comparing the table sizes between the source and target databases, I found that 30-40% of the data was missing in the target. There were no duplicates present in the source table, so that does not explain the discrepancy.
Actual behaviour:
Huge data losses in both cases, no errors in the logs
Environment info:
- Source bucket in the AWS cloud
- Destination bucket in the Azure cloud
Config:
- start: 2023-12-01T00:00:00Z,
- stop: 2024-08-31T00:00:00Z,
- batchInterval: 1h,
- option task = {
name: "Migration",
every: 30s,
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the official InfluxDB v2 cloud-to-OSS migration guide and reproduce both documented migration methods using the reported AWS source, Azure destination, and date range. Compare source and target table sizes and inspect migration logs; done means the migration preserves the source data or clearly reports any failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, azure
- Domain
- cloud, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100