OpenEuroLLM / OpenEuroLLM/training-data-packer
Different inputs for different parts
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- Avg merge
- 18h 22m
- Merged PRs (30d)
- 12
Description
Right now input source for a processing step (mode) is taken from default.input. It would be valuable to let individual parts override it.
Example: When sampling on propella we only do it on some parts for certain datasets (nemotron-cc-1.0). In the release mode it would be beneficial to say that input for the other parts not sampled are source.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how processing-step modes read default.input and how individual parts are represented and configured. Define the override behavior for sampled and unsampled parts, then verify that release mode can use the source input for parts that were not sampled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100