4paradigm / 4paradigm/OpenMLDB
Implement an efficient encoding scheme for `Traverse`
- Lingua principale
- C++
- Stelle
- 1.7k
- Fork
- 331
- Merge medio
- 12g 12h
- PR unite (30g)
- 1
Descrizione
# Problem
Currently, the records used by the `Travserse` function are encoded as follows (both pk and value are strings)
```
record length | pk length | pk | ts | value | record length | pk length | pk | ts | value ...
```
As the same pk may have multiple values, this encoding scheme may cause inefficiency because of the duplicate records for pk
# Solution
We can implement a compact format to store those data. The basic idea is to arrange the ts and values corresponding to the same pk continuously. For example, for pk1 (with ts1_x and val1_x), and pk2 (with ts2_x and val2_x):
```
pk1 | ts1_1 | val1_1 | ts1_2 | val1_2 | pk2 | ts2_1 | val2_1 | ts2_2 | val2_2
```
Specifically, considering the string type for `pk` and `val`, we need to record the length of a string as well. We also record the number of `ts` and `val` corresponding to the same pk (num_of_vals). Here is the detailed encoding scheme for one pk, more pk can be packed subsequently.
```
pk_len | pk | num_of_vals | ts | val_len | val | ts | val_len | val | ... (there are `num_of_vals` occurrences of "ts | val_len | val" in total)
```
# Related modules
https://github.com/4paradigm/OpenMLDB/blob/main/src/tablet/tablet_impl.cc#L1402
https://github.com/4paradigm/OpenMLDB/blob/main/src/base/kv_iterator.h#L81
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
The issue points to tablet_impl.cc line 1402 and kv_iterator.h line 81 as the related modules. Start by examining the current encoding in those files to understand the Traverse function's data layout. Then design and implement the new compact format as described, ensuring it handles variable-length strings correctly. Test the changes with existing database operations to verify correctness and performance improvement.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- cpp
- Ambito
- databases
- Tipo di issue
- Refactoring
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Specificata chiaramente
- Idoneità per principianti
- 45/100