php / php/php-src

token_get_all(TOKEN_PARSE): class constant named with a reserved keyword is no longer retagged to T_STRING when the same keyword appears in its initializer (regression in 8.3)

Offen
#23,552 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

@iliaal arbeitet bereits daran.

Seit 08.9.2026.

  • #23618 von @iliaal — offen
Bug Extension: tokenizer Status: Verified
Vorherrschende Sprache
C
Sterne
40.4k
Forks
8.1k
Ø Merge
2 T. 13 Std.
Gemergte PRs (30 T.)
96

Beschreibung

Description

Description

Since PHP 8.3, token_get_all($code, TOKEN_PARSE) fails to retag a reserved
keyword used as a class constant name to T_STRING when the same keyword
also appears as a class-constant fetch inside the initializer
of that constant.

The code itself parses, lints and runs fine — only the tokenizer output is wrong.

<?php
$code = '<?php class Foo { public const NEW = Bar::NEW; }';

foreach (token_get_all($code, TOKEN_PARSE) as $t) {
    if (is_array($t) && strcasecmp($t[1], 'new') === 0) {
        echo token_name($t[0]), ' [', $t[1], ']', PHP_EOL;
    }
}

Resulted in this output (PHP 8.3, 8.4, 8.5):

T_NEW [NEW]        <-- declaration: NOT retagged (bug)
T_STRING [NEW]     <-- usage after :: : retagged correctly

But I expected this output instead (and this is what PHP <= 8.2 produces):

T_STRING [NEW]
T_STRING [NEW]

https://3v4l.org/r4Yni — shows the regression starts exactly at 8.3.0 (all 8.2.x
produce the expected output; 8.3.0 through 8.5.10 produce the wrong one).

Trigger conditions

The retagging only fails when the same reserved word occurs both as the
declared constant name and as a :: constant fetch in the initializer:

Code Declared name token on 8.3+
class F { const NEW = 1; } T_STRING
class F { const NEW = Bar::BAZ; } T_STRING
class F { const FOO = Bar::NEW; } (usage) T_STRING
class F { const NEW = Bar::NEW; } T_NEW
class F { const NEW = [self::NEW]; } T_NEW

Reproduced with other semi-reserved keywords as well (the issue is not specific
to new).

Suspected cause

ext/tokenizer/tokenizer.c handles the parser feedback for
"keyword used as identifier" by searching the already-collected token list
backwards by token text and retagging the first match:

case ON_FEEDBACK: {
    HashTable *tokens_ht = ctx->tokens;
    zval *token_zv, *id_zv = NULL;
    ZEND_HASH_REVERSE_FOREACH_VAL(tokens_ht, token_zv) {
        id_zv = extract_token_id_to_replace(token_zv, text, length);
        if (id_zv) {
            break;
        }
    } ZEND_HASH_FOREACH_END();
    ...

Up to PHP 8.2 the class-constant name was reduced (and the feedback emitted)
before the initializer was scanned, so the reverse-by-text search found the
correct token. With typed class constants (PHP 8.3, #10444) the grammar has
to disambiguate the type from the constant name, the reduction of the name
happens later, and by the time the feedback fires the initializer tokens have
already been scanned — so the reverse search finds the initializer occurrence
of the same text instead of the declaration, leaving the declared name tagged
as a keyword.

PHP Version

  • PHP 8.2.33 — not affected (expected output)
  • PHP 8.3 (php:8.3-cli) — affected
  • PHP 8.4 (php:8.4-cli) — affected
  • PHP 8.5.10 (php:8.5-cli) — affected

Operating System

Reproduced on Debian-based official Docker images — the behaviour is platform
independent.

PHP Version
PHP 8.5.9 (cli) (built: Jul 28 2026 13:06:52) (NTS)
Copyright (c) The PHP Group
Built by Homebrew
Zend Engine v4.5.9, Copyright (c) Zend Technologies
    with Zend OPcache v8.5.9, Copyright (c), by Zend Technologies
Operating System

No response

Beitragsleitfaden

Beitragsleitfaden öffnen

Erste Schritte

  1. Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
  3. Forke das Repository und arbeite in einem Branch.
  4. Öffne einen Pull Request, der die Issue-Nummer nennt.

Rechercherichtung

Beginne in ext/tokenizer/tokenizer.c, insbesondere bei der Verarbeitung von ON_FEEDBACK und dem Pfad extract_token_id_to_replace(), und reproduziere anschließend das Beispiel token_get_all($code, TOKEN_PARSE) aus dem Issue. Vergleiche die Tokenfolge für class Foo { public const NEW = Bar::NEW; } zwischen PHP 8.2 und PHP 8.3+. Erledigt ist die Aufgabe, wenn sowohl das Vorkommen in der Deklaration als auch das im Initializer T_STRING erzeugen, ohne die anderen Trigger-Fälle zu regressieren.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
c, php
Bereich
compilers
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Aktiv
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
55/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.