huggingface / huggingface/course

Issue with Chapter 2 (Tokenizers/Tokenization) - output is different

Open
#100 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
MDX
Stars
4.2k
Forks
1.4k
Avg merge
13m
Merged PRs (30d)
1

Description

Hey Hugging face,

I am opening this issue because I am currently following your course (that is great by the way) and noticed some difference between the output displayed in the tokenization section and the output in my notebook.

On your lesson it looks like this
![image](https://user-images.githubusercontent.com/23313992/162052747-3ea595a6-2624-44e3-817c-cfac1b2e17fa.png)

When I am executing the same code in my EC2 instance on AWS I have that
![image](https://user-images.githubusercontent.com/23313992/162052831-eebb58ee-cca4-43ec-9140-6ede07b28122.png)

with this list `['Using', 'a', 'Trans', '##former', 'network', 'is', 'simple']` .

I am presuming that the model `bert-base-cased` could have changed since the edition of the course so it really not a big deal but it's impact the conclusion here

![image](https://user-images.githubusercontent.com/23313992/162053116-8bc7c600-a19d-4cde-8648-bd4dacf0a62d.png)

I don't know if the issue is on my side so there is the details on my current setup if you want to try to reproduce it:
* AWS ec2 p3.2xlarge
* environment miniconda Latest
* dependencies (in the requirements.txt)

[requirements.txt](https://github.com/huggingface/course/files/8430024/requirements.txt)

Don't hesitate if you have any questions

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.