huggingface / huggingface/tokenizers

How to determine the splicing logic in post_processor based on the sentence to be tokenized?

Open
#1,696 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
11k
Forks
1.2k
Avg merge
3d 8h
Merged PRs (30d)
26

Description

For example,
```python
def post_processor(self, token_ids_0, token_ids_1=None):
if "cls" in token_ids_0:
return processors.TemplateProcessing(
single=f"{cls} $A {sep}",
pair=f"{cls} $A {sep} $B {cls}",
special_tokens=[
(cls, cls_token_id),
(sep, sep_token_id),
],
)
else:
return processors.TemplateProcessing(
single=f"{sep} $A {cls}",
pair=f"{sep} $A {cls} $B {sep}",
special_tokens=[
(cls, cls_token_id),
(sep, sep_token_id),
],
)
```
Thx~

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.