Lightning-AI / Lightning-AI/litData

Try `obstore` for LitData Chunks

Open
#663 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
614
Forks
106
Avg merge
15h 8m
Merged PRs (30d)
22

Description

## 🚀 Feature
Try `obstore` for LitData Chunks
In #652 , we noticed 2-3x improvements after using `obstore`.

Notes from @tchaton :
```
So, It would be good to try it out for LitData chunks too
Instead of boto3
Maybe it would get faster
```

Contributor guide

Open the contributing guide

Research direction

Start by reviewing issue #652 and the existing LitData chunk handling that uses boto3. Try obstore for the same chunk workflow and compare its performance with the current approach. Done means the experiment has clear benchmark results and a documented decision about whether obstore improves chunk loading.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.