jupyterlite / jupyterlite/pyodide-kernel

PyodideKernel.run() leaks SyntaxError/TabError from input transformation as a raw kernel traceback (Python 3.12+)

Open
#292 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
90
Forks
47
Avg merge
3d 13h
Merged PRs (30d)
11

Description

## Summary

When a code cell contains a `SyntaxError` that CPython's tokenizer raises *during
tokenization* — most commonly `TabError: inconsistent use of tabs and spaces in
indentation` — it is not surfaced as a normal error cell. The exception escapes
`PyodideKernel.run()` unhandled, and the user sees a raw internal traceback through
`pyodide_kernel/litetransform.py` → `IPython/core/inputtransformer2.py:make_tokens_by_line`,
with no indication that they simply mixed tabs and spaces.

## Environment

- `jupyterlite-pyodide-kernel` 0.7.2
- Pyodide 0.28 (Python 3.13), IPython 9.0.2
- Also confirmed against IPython **9.14.0** (latest) — this is **not** version-specific.

## Minimal reproduction

Run this cell. The `elif`/`else` lines (and their `return`s) are indented with a TAB
character while the rest of the body uses spaces:

```python
def f(a, b):
if a:
return 1
elif b: # <- this line begins with a TAB
return 2 # <- this line begins with a TAB
```

**Actual:** a raw traceback ending in `make_tokens_by_line` →
`generate_tokens_catch_errors`.

**Expected:** a clean error cell:

```
Cell In[1], line 4
elif b:
^
TabError: inconsistent use of tabs and spaces in indentation
```

## Root cause

`PyodideKernel.run()` calls the input transforms *outside* any `try/except`:

```python
code = await self.lite_transform_manager.transform_cell(code)
exec_code = self.interpreter.transform_cell(code)
```

On Python 3.12+ the C tokenizer raises `TabError`/`IndentationError` (a `SyntaxError`
subclass) eagerly during tokenization (see python/cpython#105238). IPython's
`generate_tokens_catch_errors` only does `except tokenize.TokenError`, so it cannot
catch a `SyntaxError`; both `make_tokens_by_line` (called directly by
`litetransform.do_one_token_transform`) and `InteractiveShell.transform_cell`
propagate it.

IPython's own `InteractiveShell.run_cell` guards `transform_cell` in a `try/except`
and renders the error via `showtraceback()`/`showsyntaxerror()`. `run()` reimplements
the cell loop and omits that guard.

Note: this is **not** fixable by upgrading IPython (confirmed against 9.14.0), because
the catch clause structurally cannot catch a `SyntaxError`.

## Proposed fix

Wrap the transform calls and route to `showtraceback()` (which already dispatches
`SyntaxError` → `showsyntaxerror`), mirroring `InteractiveShell.run_cell` and the
existing `_load_packages_from_imports` handling in the same method:

```diff
--- a/pyodide_kernel/kernel.py
+++ b/pyodide_kernel/kernel.py
@@ class PyodideKernel(LoggingConfigurable):
async def run(self, code):
self.interpreter._last_traceback = None
- # apply pyodide-specific changes that need to occur before interpreting
- code = await self.lite_transform_manager.transform_cell(code)
- exec_code = self.interpreter.transform_cell(code)

results = {}

- try:
- await _load_packages_from_imports(exec_code)
- except Exception:
- self.interpreter.showtraceback()
- else:
- if self.interpreter.should_run_async(code):
- await self.interpreter.run_cell_async(code, store_history=True)
- else:
- self.interpreter.run_cell(code, store_history=True)
-
- results["payload"] = self.interpreter.payload_manager.read_payload()
- self.interpreter.payload_manager.clear_payload()
+ # apply pyodide-specific changes that need to occur before interpreting
+ try:
+ code = await self.lite_transform_manager.transform_cell(code)
+ exec_code = self.interpreter.transform_cell(code)
+ except Exception:
+ # Input transformation can raise before any cell code runs — most
+ # commonly TabError/IndentationError, which CPython's tokenizer
+ # raises eagerly on Python 3.12+ (python/cpython#105238) and which
+ # IPython's tokenizer guard (except tokenize.TokenError) cannot
+ # catch. InteractiveShell.run_cell guards transform_cell the same
+ # way; without this, the error escapes run() as an unhandled
+ # rejection and the user sees a raw kernel traceback instead of the
+ # SyntaxError. showtraceback() dispatches SyntaxError to
+ # showsyntaxerror and populates _last_traceback for the report below.
+ self.interpreter.showtraceback()
+ else:
+ try:
+ await _load_packages_from_imports(exec_code)
+ except Exception:
+ self.interpreter.showtraceback()
+ else:
+ if self.interpreter.should_run_async(code):
+ await self.interpreter.run_cell_async(code, store_history=True)
+ else:
+ self.interpreter.run_cell(code, store_history=True)
+
+ results["payload"] = self.interpreter.payload_manager.read_payload()
+ self.interpreter.payload_manager.clear_payload()

if self.interpreter._last_traceback is None:
results["status"] = "ok"
```

The error path reuses the method's existing `_last_traceback` → `results["status"] =
"error"` reporting, so the result dict stays well-formed with no new machinery. A
regression test would add the tab/space cell to the kernel test suite and assert
`status == "error"` / `ename == "TabError"` rather than a raised exception.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.