facebookresearch / facebookresearch/SONAR
Create documentation for the HF pipelines.
- Dominant language
- Python
- Stars
- 910
- Forks
- 103
- PR merge metrics
- No merged PRs in 30d
Description
We have a recently created [huggingface_pipelines](https://github.com/facebookresearch/SONAR/tree/main/huggingface_pipelines) directory with some nice code, but no obvios examples of how to use it.
One could create a documentation page that explains the purpose of the pipelines and illustrates the code with which they could be applied.
An example of the task would be to use the FLORES dataset (https://huggingface.co/datasets/facebook/flores) to compare the quality of translation from various languages to one (e.g. to English or to Spanish).
# Motivation for the task
A typical way to evaluate SONAR models for a particular language would be to encode some dataset of sentences and then to decode it to the same language (reconstruction) or to another language (translation). Then the generated texts get compared with the reference texts using numeric scores such as BLEU (from the `sacrebleu` package).
We want to use this task as an opportunity of learning more about the pipelines which are kind of glue that connects the models to the data (by e.g. batching the data to feed to the models).
# How to approach
All or most of the code elements are (probably) already somewhere in the repo, the goal is to put them together with the new Hugginface pipeline using segmentation, encoding, decoding, and BLEU computation.
A good entrypoint might be the tests (e.g. https://github.com/facebookresearch/SONAR/blob/main/tests/unit_tests/huggingface_pipelines/text.py) that illustrate some of potential use cases of the HF pipeline.
Contributor guide
Research direction
Start with tests/unit_tests/huggingface_pipelines/text.py and inspect the huggingface_pipelines directory to identify the existing segmentation, encoding, and decoding usage. Use the FLORES dataset as the example, with translation quality compared using BLEU. Done means a documentation page explains the pipeline purpose and presents a complete, understandable evaluation workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100