aws / aws/sagemaker-huggingface-inference-toolkit
[Feature Request] Support Japanese language
- Dominant language
- Python
- Stars
- 270
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
In some cases, dedicated libraries(e.g. fugashi, ipadic) are required for Japanese tokenizers.
Currently, these libraries are not included in the inference container.
Is it possible to include these libraries or to have an option in the transformers installation?
For example, if we can rewrite the Dockerfile like this, we can handle it.
`transformers[sentencepiece] ` → `transformers[ja]`
Currently, if we deploy from S3, we can work around it with `requirements.txt` and an empty `inference.py`, but if we deploy from HF Hub, we don't have a workaround.
Thanks!
Contributor guide
Research direction
Start by examining the inference container's Dockerfile and the dependency handling around requirements.txt and inference.py. Compare the existing S3 deployment workaround with the Hugging Face Hub deployment path, then verify that Japanese tokenizer dependencies are available for Hub deployments without the workaround. Done means Japanese tokenizers can be used in the inference container through the supported installation path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- dockerfile, huggingface, python
- Domain
- infrastructure, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100