jupyterlite / jupyterlite/pyodide-kernel
PyodideKernel.run() leaks SyntaxError/TabError from input transformation as a raw kernel traceback (Python 3.12+)
- Dominant language
- Python
- Stars
- 90
- Forks
- 47
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 11
Description
## Summary
When a code cell contains a `SyntaxError` that CPython's tokenizer raises *during
tokenization* — most commonly `TabError: inconsistent use of tabs and spaces in
indentation` — it is not surfaced as a normal error cell. The exception escapes
`PyodideKernel.run()` unhandled, and the user sees a raw internal traceback through
`pyodide_kernel/litetransform.py` → `IPython/core/inputtransformer2.py:make_tokens_by_line`,
with no indication that they simply mixed tabs and spaces.
## Environment
- `jupyterlite-pyodide-kernel` 0.7.2
- Pyodide 0.28 (Python 3.13), IPython 9.0.2
- Also confirmed against IPython **9.14.0** (latest) — this is **not** version-specific.
## Minimal reproduction
Run this cell. The `elif`/`else` lines (and their `return`s) are indented with a TAB
character while the rest of the body uses spaces:
```python
def f(a, b):
if a:
return 1
elif b: # <- this line begins with a TAB
return 2 # <- this line begins with a TAB
```
**Actual:** a raw traceback ending in `make_tokens_by_line` →
`generate_tokens_catch_errors`.
**Expected:** a clean error cell:
```
Cell In[1], line 4
elif b:
^
TabError: inconsistent use of tabs and spaces in indentation
```
## Root cause
`PyodideKernel.run()` calls the input transforms *outside* any `try/except`:
```python
code = await self.lite_transform_manager.transform_cell(code)
exec_code = self.interpreter.transform_cell(code)
```
On Python 3.12+ the C tokenizer raises `TabError`/`IndentationError` (a `SyntaxError`
subclass) eagerly during tokenization (see python/cpython#105238). IPython's
`generate_tokens_catch_errors` only does `except tokenize.TokenError`, so it cannot
catch a `SyntaxError`; both `make_tokens_by_line` (called directly by
`litetransform.do_one_token_transform`) and `InteractiveShell.transform_cell`
propagate it.
IPython's own `InteractiveShell.run_cell` guards `transform_cell` in a `try/except`
and renders the error via `showtraceback()`/`showsyntaxerror()`. `run()` reimplements
the cell loop and omits that guard.
Note: this is **not** fixable by upgrading IPython (confirmed against 9.14.0), because
the catch clause structurally cannot catch a `SyntaxError`.
## Proposed fix
Wrap the transform calls and route to `showtraceback()` (which already dispatches
`SyntaxError` → `showsyntaxerror`), mirroring `InteractiveShell.run_cell` and the
existing `_load_packages_from_imports` handling in the same method:
```diff
--- a/pyodide_kernel/kernel.py
+++ b/pyodide_kernel/kernel.py
@@ class PyodideKernel(LoggingConfigurable):
async def run(self, code):
self.interpreter._last_traceback = None
- # apply pyodide-specific changes that need to occur before interpreting
- code = await self.lite_transform_manager.transform_cell(code)
- exec_code = self.interpreter.transform_cell(code)
results = {}
- try:
- await _load_packages_from_imports(exec_code)
- except Exception:
- self.interpreter.showtraceback()
- else:
- if self.interpreter.should_run_async(code):
- await self.interpreter.run_cell_async(code, store_history=True)
- else:
- self.interpreter.run_cell(code, store_history=True)
-
- results["payload"] = self.interpreter.payload_manager.read_payload()
- self.interpreter.payload_manager.clear_payload()
+ # apply pyodide-specific changes that need to occur before interpreting
+ try:
+ code = await self.lite_transform_manager.transform_cell(code)
+ exec_code = self.interpreter.transform_cell(code)
+ except Exception:
+ # Input transformation can raise before any cell code runs — most
+ # commonly TabError/IndentationError, which CPython's tokenizer
+ # raises eagerly on Python 3.12+ (python/cpython#105238) and which
+ # IPython's tokenizer guard (except tokenize.TokenError) cannot
+ # catch. InteractiveShell.run_cell guards transform_cell the same
+ # way; without this, the error escapes run() as an unhandled
+ # rejection and the user sees a raw kernel traceback instead of the
+ # SyntaxError. showtraceback() dispatches SyntaxError to
+ # showsyntaxerror and populates _last_traceback for the report below.
+ self.interpreter.showtraceback()
+ else:
+ try:
+ await _load_packages_from_imports(exec_code)
+ except Exception:
+ self.interpreter.showtraceback()
+ else:
+ if self.interpreter.should_run_async(code):
+ await self.interpreter.run_cell_async(code, store_history=True)
+ else:
+ self.interpreter.run_cell(code, store_history=True)
+
+ results["payload"] = self.interpreter.payload_manager.read_payload()
+ self.interpreter.payload_manager.clear_payload()
if self.interpreter._last_traceback is None:
results["status"] = "ok"
```
The error path reuses the method's existing `_last_traceback` → `results["status"] =
"error"` reporting, so the result dict stays well-formed with no new machinery. A
regression test would add the tab/space cell to the kernel test suite and assert
`status == "error"` / `ename == "TabError"` rather than a raised exception.
Contributor guide
Assessment
This issue has not been assessed yet.