python / python/cpython

IDLE: add a token browser

Open
#152,941 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stdlib topic-IDLE type-feature
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Seeing how source is split into tokens helps in learning how Python's tokenizer works and in debugging tokenizer-dependent behavior, but IDLE offers no way to inspect the token stream.

I propose a Token Browser added as an extension on the Tools menu (Shell and editor). It shows the output of tokenize for the current window in a ttk.Treeview — one row per token with its type, string, and range.

Interaction:

  • Selecting a row highlights the corresponding source in the editor.
  • Double-clicking a row moves the editor cursor to that token.
  • The browser follows the editor: a selection selects every overlapping token row, and the cursor alone selects the single token containing it.
  • Scope is the editor selection if there is one, else the Shell's current input, else the whole window.

A PR will follow.

Linked PRs
  • gh-152946

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Python's tokenize output and IDLE's extension entry point on the Tools menu for the Shell and editor. Review how a ttk.Treeview can represent token type, string, and range, then verify the selection, double-click, cursor, and scope behavior described in the issue. Done means the browser follows the current window and maps token rows to source locations; gh-152946 is linked as the follow-up PR.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
desktop, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.