A-B test creator
Open
@Enkidu93 is already working on this.
Since Dec 8, 2023.
enhancement
onboarding
- Dominant language
- Python
- Stars
- 39
- Forks
- 7
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 5
Description
Or ABC. What is the best way to do this? Could we choose 3 experiments that have the highest Bleu score (or the ones we think are the best candidates) and auto-create an AB test to determine which of the 3 model configurations are most ideal. This could include:
- An investigation has completed runs with the pretranslations saved
- A subset of the pretranslations are used for the AB testing (book selection?)
- Either a Google sheet or potentially a Google Form is auto-created with the 3 top pretranslation sets (randomly sorted - 6 verses?) and sent to one or more translators
- Potentially both - we could create the Google Sheet and then make the form off of the sheet. The sheet could then be re-populated with the results.
- After the form is filled out, the results show back up in a Google sheet or somewhere else de-randomized
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.