Lightning-AI / Lightning-AI/litData
Add support for the reduce operator
Open
Nobody has claimed this yet.
enhancement
help wanted
won't fix
- Dominant language
- Python
- Stars
- 614
- Forks
- 106
- Avg merge
- 15h 8m
- Merged PRs (30d)
- 22
Description
🚀 Feature
Motivation
It is quite frequent users need to perform a reduce operation across a dataset. LitData could expose a custom map operator with state forwarding and automated distributed final reduction.
Pitch
Alternatives
Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by inspecting how existing map operators represent state and distribute work, then determine the API and behavior needed for state forwarding and a final distributed reduction. Done should include a defined reduce interface, distributed aggregation behavior, and coverage for the resulting operation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100