man-group / man-group/ArcticDB
Don't fragment table data on `lib.update`
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 219
- Avg merge
- 5d 13m
- Merged PRs (30d)
- 33
Description
**Is your feature request related to a problem? Please describe.**
If we do a lot of updates on small date ranges we will eventually fragment the data a lot and it will become unreadable.
I think currently if we update a single row within a table data key we will end up writing 3 table data keys.
**Describe the solution you'd like**
In some cases fragmenting the data is unavoidable if we want to have reasonable performance on small updates. E.g. if we use update with a range after the existing one. So for such cases we can have `lib.update(defragment=True)` which will pay the extra price at update time but it won't fragment table data and keep read performance.
Also I think we can decrease the fragmentation without any extra cost in cases where we split up existing table data keys.
What we do now is:
```
1. Read all table data keys which interstect the updated date range
2. Filter out the first table data key to only contain index before updated date range and write it back
3. Filter out the last table data key to only contian index after updated date range and write it back
4. Write a completely new segment with the updated date range
```
When interstacting table data keys are <3 we end up increasing the number of total segments. We can instead without extra cost write the combined segment from steps 2,3 and 4 as one table data key (and maybe split it up if it's > 100k rows)
**Describe alternatives you've considered**
Occasional defragmentation with `lib.write`
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the lib.update and lib.write entry points and trace how table data keys intersecting an update range are split and rewritten. Compare the current segment behavior with the proposed defragment=True option and the combined-segment optimization. Done means updates avoid unnecessary fragmentation while preserving the suggested handling for larger segments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100