facebookresearch / facebookresearch/SONAR

Create documentation for the HF pipelines.

Open
#41 5 comments 0 reactions 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
910
Forks
103
PR merge metrics
No merged PRs in 30d

Description

We have a recently created [huggingface_pipelines](https://github.com/facebookresearch/SONAR/tree/main/huggingface_pipelines) directory with some nice code, but no obvios examples of how to use it.

One could create a documentation page that explains the purpose of the pipelines and illustrates the code with which they could be applied.

An example of the task would be to use the FLORES dataset (https://huggingface.co/datasets/facebook/flores) to compare the quality of translation from various languages to one (e.g. to English or to Spanish).

# Motivation for the task
A typical way to evaluate SONAR models for a particular language would be to encode some dataset of sentences and then to decode it to the same language (reconstruction) or to another language (translation). Then the generated texts get compared with the reference texts using numeric scores such as BLEU (from the `sacrebleu` package).

We want to use this task as an opportunity of learning more about the pipelines which are kind of glue that connects the models to the data (by e.g. batching the data to feed to the models).

# How to approach
All or most of the code elements are (probably) already somewhere in the repo, the goal is to put them together with the new Hugginface pipeline using segmentation, encoding, decoding, and BLEU computation.

A good entrypoint might be the tests (e.g. https://github.com/facebookresearch/SONAR/blob/main/tests/unit_tests/huggingface_pipelines/text.py) that illustrate some of potential use cases of the HF pipeline.

Contributor guide

Open the contributing guide

Research direction

Start with tests/unit_tests/huggingface_pipelines/text.py and inspect the huggingface_pipelines directory to identify the existing segmentation, encoding, and decoding usage. Use the FLORES dataset as the example, with translation quality compared using BLEU. Done means a documentation page explains the pipeline purpose and presents a complete, understandable evaluation workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.