dlite-tools / dlite-tools/NLPiper

Custom Tokenizer

Open
#46 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
19
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature

Allow the user to add a custom Tokenizer with minor implementation.

**Motivation**

It is impossible to integrate every tool, but that does not mean that it shouldn't exist a CustomTokenizer that allows the user to use its own implementation.

Needs: callable function or object
Warnings: If `Compose.create_from_steps` should throw an error since it cannot load its implementation

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.