influxdata / influxdata/influxdb

1.x to 2.x migration tooling fails trying to write a new .tsm with existing filename

Open
#18,187 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/2.x
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

as per requested by @e-dard:

recurisive listing of to be migrated influxdb/data folder:
https://gist.github.com/peterpeerdeman/6e2af3f8e105ffccc181d228fb65dfc6

hello here! I'm trying my very best to migrate my 1.7.10 data to 2.0.0-beta10. both servers are not running, i've got the /root/.influxdb and /root/.influxdbv2 folders in place containing the data, but everytime I run the influxd migrate command I get an error that a certain .tsm already exists. I've double checked, the "existing" .tsm file is created earlier during the migration. This happens both when migrating all databases, as well as when migrating just one database.

@peterpeerdeman hey, could you file this as an issue and include a recursive file listing of `/root/.influxdb/data`? We only need to see the names of the TSM files in the various directories. This is probably a defect in the migration tooling. I expect that I didn't consider something and you are tickling that bug.

_Originally posted by @e-dard in https://github.com/influxdata/influxdb/pull/17016#issuecomment-632223840_

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

```
# influxd migrate --org-id 05b9f8dd8e449000
2020-05-21T16:41:26.842572Z info Resources opened {"log_id": "0Mv1cIDW000", "service": "kvstore-bolt", "path": "/root/.influxdbv2/influxd.bolt"}
Migrated /root/.influxdb/data/destiny/autogen/338/000000201-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000201-000000002.tsm
Migrated shard /root/.influxdb/data/destiny/autogen/338 to bucket 0cfef329799cb5fd in 19.4215ms
Migrated /root/.influxdb/data/destiny/autogen/356/000000337-000000003.tsm --> /root/.influxdbv2/engine/data/000000000000338-000000003.tsm
Migrated shard /root/.influxdb/data/destiny/autogen/356 to bucket 0cfef329799cb5fd in 13.7684ms
Migrated /root/.influxdb/data/destiny/autogen/374/000000327-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000329-000000002.tsm
Migrated shard /root/.influxdb/data/destiny/autogen/374 to bucket 0cfef329799cb5fd in 14.6298ms
Migrated /root/.influxdb/data/destiny/autogen/392/000000336-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000339-000000002.tsm
Migrated shard /root/.influxdb/data/destiny/autogen/392 to bucket 0cfef329799cb5fd in 23.4419ms
Migrated /root/.influxdb/data/destiny/autogen/410/000000337-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000341-000000002.tsm
Migrated shard /root/.influxdb/data/destiny/autogen/410 to bucket 0cfef329799cb5fd in 25.7042ms
Migrated /root/.influxdb/data/destiny/autogen/428/000000139-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000144-000000002.tsm
Migrated shard /root/.influxdb/data/destiny/autogen/428 to bucket 0cfef329799cb5fd in 14.0162ms
Migrated /root/.influxdb/data/doorbell/autogen/240/000000003-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000009-000000002.tsm
Migrated shard /root/.influxdb/data/doorbell/autogen/240 to bucket e5683265f2e44abd in 6.1435ms
Migrated /root/.influxdb/data/doorbell/autogen/251/000000003-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000010-000000002.tsm
Migrated shard /root/.influxdb/data/doorbell/autogen/251 to bucket e5683265f2e44abd in 6.8534ms
Migrated /root/.influxdb/data/doorbell/autogen/268/000000005-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000013-000000002.tsm
Migrated shard /root/.influxdb/data/doorbell/autogen/268 to bucket e5683265f2e44abd in 24.9904ms
Migrated /root/.influxdb/data/doorbell/autogen/285/000000002-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000011-000000002.tsm
Migrated shard /root/.influxdb/data/doorbell/autogen/285 to bucket e5683265f2e44abd in 6.3018ms
Migrated /root/.influxdb/data/fitbit/autogen/7/000000001-000000001.tsm --> /root/.influxdbv2/engine/data/000000000000011-000000001.tsm
Migrated shard /root/.influxdb/data/fitbit/autogen/7 to bucket 36a8dd4f954efd8a in 7.596ms
Migrated /root/.influxdb/data/fitbit/autogen/9/000000003-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000014-000000002.tsm
Migrated shard /root/.influxdb/data/fitbit/autogen/9 to bucket 36a8dd4f954efd8a in 24.3645ms
Migrated /root/.influxdb/data/fitbit/autogen/23/000000005-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000017-000000002.tsm
Migrated shard /root/.influxdb/data/fitbit/autogen/23 to bucket 36a8dd4f954efd8a in 28.7097ms
Migrated /root/.influxdb/data/fitbit/autogen/224/000000006-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000019-000000002.tsm
Migrated shard /root/.influxdb/data/fitbit/autogen/224 to bucket 36a8dd4f954efd8a in 24.5256ms
Migrated /root/.influxdb/data/fitbit/autogen/242/000000015-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000029-000000002.tsm
Migrated shard /root/.influxdb/data/fitbit/autogen/242 to bucket 36a8dd4f954efd8a in 27.7407ms
Migrated /root/.influxdb/data/fitbit/autogen/259/000000017-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000032-000000002.tsm
Migrated shard /root/.influxdb/data/fitbit/autogen/259 to bucket 36a8dd4f954efd8a in 25.9605ms
Migrated /root/.influxdb/data/fitbit/autogen/276/000000007-000000002.tsm --> /root/.influxdbv2/engine/data/000000000000023-000000002.tsm
Migrated shard /root/.influxdb/data/fitbit/autogen/276 to bucket 36a8dd4f954efd8a in 22.9052ms
Error: error processing TSM file "/root/.influxdb/data/fitbit/autogen/293/000000002-000000002.tsm": operation not allowed, file "/root/.influxdbv2/engine/data/000000000000019-000000002.tsm" exists
```
__Expected behavior:__
succesful migration

__Actual behavior:__
error as seen above

__Environment info:__
* System info: Linux 4.19.76-linuxkit x86_6
* InfluxDB version: InfluxDB 2.0.0-beta.10 (git: d00706dea8) build_date: 2020-05-21T17:04:04Z

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing `influxd migrate --org-id 05b9f8dd8e449000` with the listed `/root/.influxdb/data` and `/root/.influxdbv2/engine/data` contents, focusing on the conflicting `.tsm` paths. Trace the migration entry point and verify that a complete migration succeeds without rejecting a destination filename that was created earlier.

Written by the indexing model from the issue text.

Assessment

Domain
databases, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.