googleforgames / googleforgames/clean-chat
Add support for multiple languages when processing text
- Dominant language
- Python
- Stars
- 54
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
This is a new feature request and not related to a bug or issue within the current code. However, English is the only supported language and we would like to enable multilingual support.
**Describe the solution you'd like**
Add multiple language support for text via the Google Cloud Translation API. The proposed solution will process data in the following order:
1. Capture raw text input from the user. [this step already exists as part of Clean Chat]
2. Use the Google Cloud Translation API to auto-detect the language.
3. If language is not equal to English, then convert the non-English language to English using the Cloud Translation API.
4. Score the translated text for toxicity as part of the Clean Chat pipeline. [this step already exists as part of Clean Chat]
**Describe alternatives you've considered**
I've considered the tradeoffs between using the Translation API versus analyzing the native language(s) directly. While analyzing the native language directly may produce more accurate results, it does require additional model training and complexity in maintaining multiple language models. Native language analysis may be a feature that we add in the future, but our results with the Google Cloud Translation API look promising and the API is stable, reliable, and scalable.
**Additional context**
Need to add a flag for users to enable or disable multiple language support. This feature only applies to text input.
Contributor guide
Assessment
This issue has not been assessed yet.