4paradigm / 4paradigm/OpenMLDB
Implement an efficient encoding scheme for `Traverse`
- Dominant language
- C++
- Stars
- 1.7k
- Forks
- 331
- Avg merge
- 12d 12h
- Merged PRs (30d)
- 1
Description
# Problem
Currently, the records used by the `Travserse` function are encoded as follows (both pk and value are strings)
```
record length | pk length | pk | ts | value | record length | pk length | pk | ts | value ...
```
As the same pk may have multiple values, this encoding scheme may cause inefficiency because of the duplicate records for pk
# Solution
We can implement a compact format to store those data. The basic idea is to arrange the ts and values corresponding to the same pk continuously. For example, for pk1 (with ts1_x and val1_x), and pk2 (with ts2_x and val2_x):
```
pk1 | ts1_1 | val1_1 | ts1_2 | val1_2 | pk2 | ts2_1 | val2_1 | ts2_2 | val2_2
```
Specifically, considering the string type for `pk` and `val`, we need to record the length of a string as well. We also record the number of `ts` and `val` corresponding to the same pk (num_of_vals). Here is the detailed encoding scheme for one pk, more pk can be packed subsequently.
```
pk_len | pk | num_of_vals | ts | val_len | val | ts | val_len | val | ... (there are `num_of_vals` occurrences of "ts | val_len | val" in total)
```
# Related modules
https://github.com/4paradigm/OpenMLDB/blob/main/src/tablet/tablet_impl.cc#L1402
https://github.com/4paradigm/OpenMLDB/blob/main/src/base/kv_iterator.h#L81
Contributor guide
Research direction
The issue points to tablet_impl.cc line 1402 and kv_iterator.h line 81 as the related modules. Start by examining the current encoding in those files to understand the Traverse function's data layout. Then design and implement the new compact format as described, ensuring it handles variable-length strings correctly. Test the changes with existing database operations to verify correctness and performance improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100