Commit ff4959b for pdb colorization GH-113081 broke some pdb backtraces

Bug report

Bug description:

If you put a breakpoint here:

diff --git a/Lib/tokenize.py b/Lib/tokenize.py
index 117b485b934..9d215ea4ea1 100644
--- a/Lib/tokenize.py
+++ b/Lib/tokenize.py
@@ -574,6 +574,7 @@ def _transform_msg(msg):

 def _generate_tokens_from_c_tokenizer(source, encoding=None, extra_tokens=False):
     """Tokenize a source reading Python code as unicode strings using the internal C tokenizer"""
+    breakpoint()
     if encoding is None:
         it = _tokenize.TokenizerIter(source, extra_tokens=extra_tokens)
     else:

and run this:

./python.exe  -m test test_tokenize -m test_random_files -uall -v
❯ ./python.exe  -m test test_tokenize -m test_random_files -uall -v
== CPython 3.14.0a7+ (bisect/good-2bbcaedb75942389dacb51866948f40de5951c9c-4-g08d7687094c-dirty:08d768) [Clang 17.0.0 (clang-1700.6.3.2)]
== macOS-26.2-arm64-arm-64bit-Mach-O little-endian
== Python build: release
== cwd: /Users/pablogsal/github/python/3.14/build/test_python_worker_11047æ
== CPU count: 10
== encodings: locale=UTF-8 FS=utf-8
== resources: all

Using random seed: 1175774464
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 4.01 Run 1 test sequentially in a single process
0:00:00 load avg: 4.01 [1/1] test_tokenize
> /Users/pablogsal/github/python/3.14/Lib/tokenize.py(577)_generate_tokens_from_c_tokenizer()
-> breakpoint()
(Pdb) c
test_random_files (test.test_tokenize.TestRoundtrip.test_random_files) ... > /Users/pablogsal/github/python/3.14/Lib/tokenize.py(577)_generate_tokens_from_c_tokenizer()
-> breakpoint()
(Pdb)
> /Users/pablogsal/github/python/3.14/Lib/tokenize.py(577)_generate_tokens_from_c_tokenizer()
-> breakpoint()
(Pdb) bt
(Pdb)

pdb cannot show the stack. Bisecting points to

ff4959b6b0cf6da5d2f59260a9103b31c812b0ba is the first bad commit (for #113081 / #133355)
commit ff4959b6b0cf6da5d2f59260a9103b31c812b0ba (HEAD)
Author: Tian Gao gaogaotiantian@hotmail.com
Date: Mon May 5 00:49:52 2025 -0700

gh-113081: Highlight source code in pdb (#133355)

Doc/library/pdb.rst | 8 +++-
Doc/whatsnew/3.14.rst | 5 ++
Lib/pdb.py | 31 +++++++++---
Lib/test/test_pdb.py | 37 +++++++++++++++
...-18-48-54.gh-issue-113081.JsLJ1X.rst | 1 +
5 files changed, 74 insertions(+), 8 deletions(-)
create mode 100644 Misc/NEWS.d/next/Library/2025-05-03-18-48-54.gh-issue-113081.JsLJ1X.rst

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS