huggingface / huggingface/tokenizers
How to determine the splicing logic in post_processor based on the sentence to be tokenized?
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
For example,
```python
def post_processor(self, token_ids_0, token_ids_1=None):
if "cls" in token_ids_0:
return processors.TemplateProcessing(
single=f"{cls} $A {sep}",
pair=f"{cls} $A {sep} $B {cls}",
special_tokens=[
(cls, cls_token_id),
(sep, sep_token_id),
],
)
else:
return processors.TemplateProcessing(
single=f"{sep} $A {cls}",
pair=f"{sep} $A {cls} $B {sep}",
special_tokens=[
(cls, cls_token_id),
(sep, sep_token_id),
],
)
```
Thx~
Contributor guide
Assessment
This issue has not been assessed yet.