IDLE: add a token browser
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
Seeing how source is split into tokens helps in learning how Python's tokenizer works and in debugging tokenizer-dependent behavior, but IDLE offers no way to inspect the token stream.
I propose a Token Browser added as an extension on the Tools menu (Shell and editor). It shows the output of tokenize for the current window in a ttk.Treeview — one row per token with its type, string, and range.
Interaction:
- Selecting a row highlights the corresponding source in the editor.
- Double-clicking a row moves the editor cursor to that token.
- The browser follows the editor: a selection selects every overlapping token row, and the cursor alone selects the single token containing it.
- Scope is the editor selection if there is one, else the Shell's current input, else the whole window.
A PR will follow.
Linked PRs
- gh-152946
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Python's tokenize output and IDLE's extension entry point on the Tools menu for the Shell and editor. Review how a ttk.Treeview can represent token type, string, and range, then verify the selection, double-click, cursor, and scope behavior described in the issue. Done means the browser follows the current window and maps token rows to source locations; gh-152946 is linked as the follow-up PR.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- desktop, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 30/100