token_get_all(TOKEN_PARSE): class constant named with a reserved keyword is no longer retagged to T_STRING when the same keyword appears in its initializer (regression in 8.3)
- 主要言語
- C
- スター
- 40.4k
- フォーク
- 8.1k
- 平均マージ
- 2日 13時間
- マージ済み PR(30日)
- 96
説明
Description
Description
Since PHP 8.3, token_get_all($code, TOKEN_PARSE) fails to retag a reserved
keyword used as a class constant name to T_STRING when the same keyword
also appears as a class-constant fetch inside the initializer of that constant.
The code itself parses, lints and runs fine — only the tokenizer output is wrong.
<?php
$code = '<?php class Foo { public const NEW = Bar::NEW; }';
foreach (token_get_all($code, TOKEN_PARSE) as $t) {
if (is_array($t) && strcasecmp($t[1], 'new') === 0) {
echo token_name($t[0]), ' [', $t[1], ']', PHP_EOL;
}
}
Resulted in this output (PHP 8.3, 8.4, 8.5):
T_NEW [NEW] <-- declaration: NOT retagged (bug)
T_STRING [NEW] <-- usage after :: : retagged correctly
But I expected this output instead (and this is what PHP <= 8.2 produces):
T_STRING [NEW]
T_STRING [NEW]
https://3v4l.org/r4Yni — shows the regression starts exactly at 8.3.0 (all 8.2.x
produce the expected output; 8.3.0 through 8.5.10 produce the wrong one).
Trigger conditions
The retagging only fails when the same reserved word occurs both as the
declared constant name and as a :: constant fetch in the initializer:
| Code | Declared name token on 8.3+ |
|---|---|
class F { const NEW = 1; } |
T_STRING ✅ |
class F { const NEW = Bar::BAZ; } |
T_STRING ✅ |
class F { const FOO = Bar::NEW; } |
(usage) T_STRING ✅ |
class F { const NEW = Bar::NEW; } |
T_NEW ❌ |
class F { const NEW = [self::NEW]; } |
T_NEW ❌ |
Reproduced with other semi-reserved keywords as well (the issue is not specific
to new).
Suspected cause
ext/tokenizer/tokenizer.c handles the parser feedback for
"keyword used as identifier" by searching the already-collected token list
backwards by token text and retagging the first match:
case ON_FEEDBACK: {
HashTable *tokens_ht = ctx->tokens;
zval *token_zv, *id_zv = NULL;
ZEND_HASH_REVERSE_FOREACH_VAL(tokens_ht, token_zv) {
id_zv = extract_token_id_to_replace(token_zv, text, length);
if (id_zv) {
break;
}
} ZEND_HASH_FOREACH_END();
...
Up to PHP 8.2 the class-constant name was reduced (and the feedback emitted)
before the initializer was scanned, so the reverse-by-text search found the
correct token. With typed class constants (PHP 8.3, #10444) the grammar has
to disambiguate the type from the constant name, the reduction of the name
happens later, and by the time the feedback fires the initializer tokens have
already been scanned — so the reverse search finds the initializer occurrence
of the same text instead of the declaration, leaving the declared name tagged
as a keyword.
PHP Version
- PHP 8.2.33 — not affected (expected output)
- PHP 8.3 (
php:8.3-cli) — affected - PHP 8.4 (
php:8.4-cli) — affected - PHP 8.5.10 (
php:8.5-cli) — affected
Operating System
Reproduced on Debian-based official Docker images — the behaviour is platform
independent.
PHP Version
PHP 8.5.9 (cli) (built: Jul 28 2026 13:06:52) (NTS)
Copyright (c) The PHP Group
Built by Homebrew
Zend Engine v4.5.9, Copyright (c) Zend Technologies
with Zend OPcache v8.5.9, Copyright (c), by Zend Technologies
Operating System
No response
コントリビューションガイド
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
調査の方向性
ext/tokenizer/tokenizer.c から始め、特に ON_FEEDBACK の処理と extract_token_id_to_replace() の経路を確認してから、issue にある token_get_all($code, TOKEN_PARSE) の例を再現します。PHP 8.2 と PHP 8.3+ で、class Foo { public const NEW = Bar::NEW; } のトークン列を比較します。宣言側と初期化子側の両方の出現箇所が T_STRING を生成し、他のトリガーケースをリグレッションさせないことが完了条件です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- c, php
- 領域
- compilers
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 活発
- 明瞭さ
- 明確に書かれている
- 初心者へのやさしさ
- 55/100