OpenEuroLLM / OpenEuroLLM/training-data-packer
Use annotations from metadata
Open
Nobody has claimed this yet.
idea
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- Avg merge
- 18h 22m
- Merged PRs (30d)
- 12
Description
Use annotations from metadata to control what to process.
Example of use cases:
- Select desired method of pii (provide ability to alter between regex and openai-privacy, etc.)
- Automatically detect and fail if indata is not avaliable. For instance if there is no PII or contamination files, right now it is run without doing anything except provide a log message. Here we can fail if directory structure not match metadata.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how metadata annotations are read and how the packer selects processing steps and validates input directories. Clarify the annotation format and supported processing methods before changing behavior. Done means the selected method is controlled by metadata and missing required data causes a failure rather than only a log message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100