OpenEuroLLM / OpenEuroLLM/training-data-packer
More balanced sharding
Open
Nobody has claimed this yet.
idea
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- Avg merge
- 18h 22m
- Merged PRs (30d)
- 12
Description
It would be valuable to have a more balanced sharding to make workers to run about same amount of time.
One solution:
Write a submitter script
- list all files
- Remove files already done.
- group them into n-shards looking on size
- Create an indata file that is consumed by packager.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the packager's indata interface and the existing way files are listed and marked done. Implement the proposed submitter flow: remove completed files, group the remaining files into size-aware shards, and create an indata file consumed by the packager; done means workers receive more balanced workloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100