4paradigm / 4paradigm/OpenMLDB

disk table supports duplicate key?

Open
#2,318 0 comments 0 reactions 1 assignee Claimed by @zhanghaohit View on GitHub
enhancement storage-engine
Dominant language
C++
Stars
1.7k
Forks
331
Avg merge
12d 12h
Merged PRs (30d)
1

Description

**Describe the feature you'd like**

One big difference between disk table and mem table is that disk table only allows one unique key per index.
For example, if the index is . the rows we inserted are:

|id |col1 | col2|
|-------| ------ | ----- |
1 | k1 | 1
2 | k1 | 1
3 | k2 | 1
4 | k2 | 2

In mem table, it will be:
|id |col1 | col2|
|-------| ------ | ----- |
1 | k1 | 1
2 | k1 | 1
3 | k2 | 1
4 | k2 | 2

However, in disk table, it will be:
|id |col1 | col2|
|-------| ------ | ----- |
1 | k1 | 1
3 | k2 | 1
4 | k2 | 2

because row 1 and row 2 have the same value of .

We should discuss whether we should support the same semantics for both disk table and memory table.

**Additional context**
One possible solution is to add a sequence number to the combined key for disk table.

Contributor guide

Open the contributing guide

Research direction

Look at the disk table and memory table implementations to understand how keys are handled. The issue suggests adding a sequence number to the combined key for disk tables to allow duplicates. Start by examining the index structures and insertion logic, then design a change that maintains performance while supporting duplicate keys. Testing will involve verifying that duplicate rows are preserved in disk tables as they are in memory tables.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.