dlite-tools / dlite-tools/NLPiper
Custom Tokenizer
Open
enhancement
- Dominant language
- Python
- Stars
- 19
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## 🚀 Feature
Allow the user to add a custom Tokenizer with minor implementation.
**Motivation**
It is impossible to integrate every tool, but that does not mean that it shouldn't exist a CustomTokenizer that allows the user to use its own implementation.
Needs: callable function or object
Warnings: If `Compose.create_from_steps` should throw an error since it cannot load its implementation
Contributor guide
Assessment
This issue has not been assessed yet.