guyluz11 / guyluz11/flutter_local_summarizer
Support capital letters
Open
- Dominant language
- C++
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
The tokenizer has a different output than the one in Python.
For the text "(CNN)When"
Result [75, 29, 29, 15832]
Should be [41, 254, 17235, 61, 10555, 1]
Dart code
```dart
String getLongText = """(CNN)When""";
Uint32List encodedList = tiktoken.encode(getLongText);
print('encodedList');
print(encodedList);
```
Python code
```Python
ARTICLE = """(CNN)When"""
inputs = tokenizer(ARTICLE, return_tensors="pt", padding=True, truncation=True)
print(inputs['input_ids'])
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.