apache / apache/kvrocks

Duplicate data was generated during expansion.

Open
#2,723 10 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
4.4k
Forks
658
Avg merge
1d 20h
Merged PRs (30d)
10

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.

### Version

2.10

### Minimal reproduce step

Env: Kvrocks cluster with 4 shards.

1. Using go-redis and opening multiple goroutines to concurrently generate data continuously.

``` golang
userId := helper.GetRandomInt64InRange(1, 10000)
key := strconv.FormatInt(userId, 10)

uuidStr := uuid.New().String()
timestamp := time.Now().UnixNano()
value := fmt.Sprintf("%s-%d", uuidStr, timestamp)

_, err := c.Do(context.TODO(), 2, "LPUSH", key, value)
```

2. Use the following command to expand the Kvrocks cluster from 4 shards to 8 shards.

```
CLUSTERX SETNODEID $NODE_ID $NODE_ID
CLUSTERX MIGRATE $slot $dst_nodeid
CLUSTERX SETSLOT $slot NODE $node_id $new_version
```

3. Check data after expand.

### What did you expect to see?

no duplicate data was generated.

### What did you see instead?

``` shell
[root@iZwz9dff7pa0adxvbwkbc8Z ~]# ./check_redis_list
2025/01/16 13:34:20 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
LIST 1978 has duplicate data:
duplicate data: 99b22bda-96de-43ec-98d4-d74d5a590d11-1736994529584945168
duplicate data index: [43 44]
list length: 395

LIST 3171 has duplicate data:
duplicate data: 9bb7a7cc-3004-4d5b-965d-fe171badc21a-1736994509584166314
duplicate data index: [86 87]
list length: 412

LIST 3468 has duplicate data:
duplicate data: 9c3bc47b-d92c-4d55-b588-9fbc594849fc-1736994519583905106
duplicate data index: [65 66]
list length: 406

LIST 3616 has duplicate data:
duplicate data: 41342d98-ace0-4212-b9ea-437772a17e68-1736994539585064098
duplicate data index: [32 33]
list length: 376

LIST 5519 has duplicate data:
duplicate data: c9fb599e-453e-4615-bc7c-74b1b8a8bec0-1736994529583967377
duplicate data index: [51 52]
list length: 414

LIST 6445 has duplicate data:
duplicate data: 3d1c3404-af08-4d47-a0f8-2fa0cea0885a-1736994509583942704
duplicate data index: [87 88]
list length: 377

LIST 8973 has duplicate data:
duplicate data: ab937bce-f43f-4f2c-a65a-b1c6cc5f08b9-1736994529584052504
duplicate data index: [55 56]
list length: 414

LIST 9250 has duplicate data:
duplicate data: 1550b5fc-ba5a-4d88-9b9d-daf9c52650ef-1736994509583124277
duplicate data index: [82 83]
list length: 421

LIST 9555 has duplicate data:
duplicate data: 606ab92e-4052-43fb-900d-91e45892ef2a-1736994529583492941
duplicate data index: [59 60]
list length: 395

```

### Anything Else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the report with concurrent LPUSH operations on a four-shard cluster while expanding it to eight shards using the listed CLUSTERX commands. Inspect the cluster expansion and slot-migration entry points implicated by that sequence, and verify that migrated lists contain no duplicate values afterward.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, redis
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.