guyluz11 / guyluz11/flutter_local_summarizer

Support capital letters

Open
#7 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

The tokenizer has a different output than the one in Python.

For the text "(CNN)When"

Result [75, 29, 29, 15832]

Should be [41, 254, 17235, 61, 10555, 1]

Dart code
```dart
String getLongText = """(CNN)When""";
Uint32List encodedList = tiktoken.encode(getLongText);

print('encodedList');
print(encodedList);
```

Python code
```Python
ARTICLE = """(CNN)When"""
inputs = tokenizer(ARTICLE, return_tensors="pt", padding=True, truncation=True)
print(inputs['input_ids'])
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.