huggingface / huggingface/course

Code error in ch 3, Processing the data

Open
#1,181 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
MDX
Stars
4.2k
Forks
1.4k
Avg merge
13m
Merged PRs (30d)
1

Description

Hi everyone, I was following up the whole tutorial about LLM course and stuck by something.

# Problem

There's a code snippet about [initializing tokenizer with the dataset](https://huggingface.co/learn/llm-course/en/chapter3/2#preprocessing-a-dataset) from *MPRC*:

```py
tokenized_dataset = tokenizer(
raw_datasets["train"]["sentence1"],
raw_datasets["train"]["sentence2"],
padding=True,
truncation=True,
)
```
As the tutorial suggests, it should work well but I've encountered the following error:
```py
Traceback (most recent call last):
File "/Users/leepil/projects/hf/main.py", line 10, in
tokenized_dataset = tokenizer(
raw_datasets["train"]["sentence1"],
...<2 lines>...
truncation=True,
)
File "/Users/leepil/projects/hf/.venv/lib/python3.14/site-packages/transformers/tokenization_utils_base.py", line 2559, in __call__
encodings = self._encode_plus(
text=text,
...<4 lines>...
**all_kwargs,
)
File "/Users/leepil/projects/hf/.venv/lib/python3.14/site-packages/transformers/tokenization_utils_tokenizers.py", line 799, in _encode_plus
raise ValueError(
...<2 lines>...
)
ValueError: text input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) or `list[list[str]]` (batch of pretokenized examples) or `list[tuple[list[str], list[str]]]` (batch of pretokenized sequence pairs).
```

After investigating the error, `PreTrainedTokenizerBase` in`transformers/tokenization_utils_base.py` only accept the input which is type of:
```py
text: TextInput | PreTokenizedInput | list[TextInput] | list[PreTokenizedInput] | None = None,
```

And apparently the type of `raw_datasets["train"]["sentence1"]` is `datasets.arrow_dataset.Dataset`.

# Suggestion
Because of that, wouldn't it be better to cover the input arguments with `list()` so it can be called properly like:
```py
tokenized_dataset = tokenizer(
list(raw_datasets["train"]["sentence1"]),
list(raw_datasets["train"]["sentence2"]),
padding=True,
truncation=True,
)
```
This code gives me a output what I expected. I'm very new to this library so please let me know if there's something that I missed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Open the linked Chapter 3 “Preprocessing a dataset” section and reproduce the tokenizer call with the shown MPRC dataset columns. Check the column value types against the tokenizer’s accepted inputs, then verify that the documented example runs successfully and matches the intended output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.