Lightning-AI / Lightning-AI/litData
Try `obstore` for LitData Chunks
Open
enhancement
- Dominant language
- Python
- Stars
- 614
- Forks
- 106
- Avg merge
- 15h 8m
- Merged PRs (30d)
- 22
Description
## 🚀 Feature
Try `obstore` for LitData Chunks
In #652 , we noticed 2-3x improvements after using `obstore`.
Notes from @tchaton :
```
So, It would be good to try it out for LitData chunks too
Instead of boto3
Maybe it would get faster
```
Contributor guide
Research direction
Start by reviewing issue #652 and the existing LitData chunk handling that uses boto3. Try obstore for the same chunk workflow and compare its performance with the current approach. Done means the experiment has clear benchmark results and a documented decision about whether obstore improves chunk loading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100