huggingface / huggingface/diffusers

[Bug] ChatGLMTokenizer special token property getters and setters are broken

Open Beginner friendly
#14,291 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

### Describe the bug

In `ChatGLMTokenizer` (`src/diffusers/pipelines/kolors/tokenizer.py`), the property getters for `pad_token`, `unk_token`, and `eos_token` hardcode return values (e.g. `pad_token` returns `""` instead of `""`). Furthermore, assigning new special token values (e.g. `tokenizer.pad_token = ""`) has no effect on the property getter because the custom setters update private attributes (`_pad_token`) while the getters ignore them or use hardcoded strings.

### Reproduction

```python
from diffusers.pipelines.kolors import ChatGLMTokenizer

# Instantiating ChatGLMTokenizer or viewing properties
# tokenizer.pad_token returns '' instead of ''
# Setting tokenizer.pad_token = '' does not update tokenizer.pad_token
```

### System Info

- Diffusers version: 0.36.0.dev0
- Python version: 3.11
- PyTorch version: 2.x

### Who can help?

@yiyixuxu @asomoza @DN6

Contributor guide

Open the contributing guide

Research direction

Open src/diffusers/pipelines/kolors/tokenizer.py and inspect ChatGLMTokenizer's pad_token, unk_token, and eos_token properties. Reproduce the reported default values and assignments such as tokenizer.pad_token = "". Done means the getters return the correct defaults and reflect values assigned through their setters.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.