QuantEcon / QuantEcon/QuantEcon.py

WASM: cache=True functions that link a cache-restored callee fail with "no compiled object yet" (simplex_grid → num_compositions_jit → comb_jit)

Open
#944 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

maintenance
Dominant language
Python
Stars
2.4k
Forks
2.3k
Avg merge
3d 3h
Merged PRs (30d)
3

Description

[!NOTE]
Updated 2026-08-21. The original body attributed the failure to comb_jit's eager signature and said the Emscripten build does not support the cache's object path, so the error fired at definition; the upstream analysis in emscripten-forge/recipes#6309 shows the real mechanism is a cache-miss cache=True caller linking a cache-hit callee in the same warm session, so the signature only decides which function trips it first. The mechanism paragraph, the Options table (now including what PR #943 does) and the Tasks are corrected below; the traceback, reproducer, upstream link and the 2026-08-21 decision are unchanged. The original text is preserved in the edit history.

Part of #925 (Phase 1). Surfaced while reviewing #942; split out of #929, which is now documentation-only.

Problem

On the emscripten-forge Numba build running in the xeus-python JupyterLite kernel, calling simplex_grid fails before any computation happens:

File /lib/python3.13/site-packages/numba/core/codegen.py:632, in CodeLibrary._get_compiled_object(self)
    630     raise ValueError("object caching not enabled in %s" % (self,))
    631 if self._compiled_object is None:
--> 632     raise RuntimeError("no compiled object yet for %s" % (self,))
    633 return self._compiled_object

RuntimeError: no compiled object yet for <Library 'comb_jit' at 0x9e12360>

Reproducer (in the browser kernel): import quantecon as qe; qe.simplex_grid(3, 4). Confirmed independently by @kp992 and @oyamad on #942. Precondition: the persistent cache must be warmcomb_jit cached by an earlier session on the same browser profile (any session that did import quantecon is enough) while num_compositions_jit is not. A cold single-session kernel does not reproduce it; the upstream control is pointing NUMBA_CACHE_DIR at an empty directory.

Mechanism (see emscripten-forge/recipes#6309). The emscripten-forge build does persist the Numba cache — patch 0006 enables object caching and writes to /drive/.cache/numba — but its save path is what fails. When a cache=True function is saved, patch 0006's _serialize_wasm_linking_libraries walks the libraries it links and calls _get_compiled_object() on each. A library that was restored from the cache no longer has that object: mainline Numba's _object_getbuffer_hook hands the buffer to the engine and sets _compiled_object = None. So the crash occurs exactly when a function compiled fresh in this session (cache miss) links a callee restored from cache in the same session (cache hit); fresh+fresh and hit+hit are both fine. The error therefore fires at the first call of the caller, not at definition — in the traceback above it is simplex_gridDispatcher.compilesave_overload, i.e. the save of the freshly compiled num_compositions_jit, whose linked comb_jit was restored at import. The failure is self-perpetuating: the save that would cache the caller is the operation that crashes, so the caller can never become a cache hit and the crash recurs in every later session.

comb_jit is the deterministic trigger only because it is the library's one function with an eager signature and cache=True together (@jit(types.intp(types.intp, types.intp), nopython=True, cache=True)): it is compiled at every import quantecon, so in every warm session it is restored from cache before any caller has a chance to be compiled fresh alongside it. The eager signature is not itself the bug — the upstream MWE crashes two lazy @njit(cache=True) functions f/g the same way. Lazy cache=True functions with no jitted callees (e.g. _cartesian_index) are unaffected.

This, not integer width, is what currently blocks simplex_grid, num_compositions_jit and k_array_rank_jit in the browser. The cache=Truecache=True chains on main are:

  • simplex_gridnum_compositions_jitcomb_jit — fails deterministically on a warm cache (this issue).
  • k_array_rank_jitcomb_jit — same trigger, same outcome.
  • _lemke_howson_tbl (game_theory/lemke_howson.py), linprog_simplex and lcp_lemke_pivoting / _lex_min_ratio_test — latently exposed: the same crash whenever one caller cached the shared helpers in an earlier session and a different, not-yet-cached caller is compiled in a later one (e.g. linprog_simplex in session 1, then a first call to lcp_lemke in session 2).

Upstream

Filed by @oyamad as emscripten-forge/recipes#6309 (2026-08-20; open, two follow-up comments as of 2026-08-21). The report gives the root cause above and a quantecon-free f/g MWE; the follow-ups add a copy-paste three-cell repro and verify the first candidate patch (retain _compiled_object after restore) by monkeypatching a live kernel: simplex_grid(3, 4) completes and writes the num_compositions_jit / simplex_grid entries, which unpatched kernels afterwards load successfully. No upstream PR yet. The report also notes a secondary issue — cache writes shortly before a page reload can be lost because /drive is backed by the asynchronous Contents API — which makes the crash intermittent to reproduce by hand.

Options

Option Effect Cost
Wait for the upstream fix No library change simplex_grid stays broken in the browser until the recipe is rebuilt
Drop cache=True from comb_jit only Works today for the three comb_jit chains: comb_jit is then compiled fresh at import, never restored, and skipped at restore time by the is_symbol_defined check; comb_jit is tiny so the recompile cost is negligible Loses disk caching for one function natively; leaves the _pivoting / _lex_min_ratio_test chains exposed
Gate cache on sys.platform != "emscripten" Native behaviour unchanged Adds a platform branch to a decorator; same coverage as the row above unless applied to every cache=True function, which would give up persistent caching in the browser altogether
Drop the eager signature only (what PR #943 for #930 does) comb_jit no longer compiles at import, so a cold session that calls simplex_grid first saves the whole chain fresh and later sessions are all hits Does not fix this issue — a lazily cached comb_jit is still restored in later sessions, so the next not-yet-cached caller (k_array_rank_jit, or simplex_grid / num_compositions_jit when their entries are missing) fails identically; it narrows the exposure to warm sessions in which comb_jit was cached by a different caller

Decision (2026-08-21): wait for the upstream fix; no library-side change for now. The candidate patch is already verified in emscripten-forge/recipes#6309, so the remaining wait is for an upstream PR and a recipe rebuild. Revisit if emscripten-forge/recipes#6309 stalls.

Tasks

  • Track emscripten-forge/recipes#6309 (open; candidate patch verified by monkeypatch, no upstream PR as of 2026-08-21)
  • Decide library-side workaround — wait for upstream (see above)
  • Regression test: the reproducer already ships as test_simplex_grid in ci/wasm/smoke_test.py (PR #938, merged 2026-08-21), but a cold single-session run passes, so it catches nothing as it stands. The #933 browser job needs to run it against a warm cache — a second kernel session on the same browser profile, or a pre-seeded /drive/.cache/numba containing only the comb_jit entry — and list it as an expected failure until upstream ships
  • Fix the comment above test_simplex_grid in ci/wasm/smoke_test.py: it still cites #929 (intp boundary); it should cite this issue and note the warm-cache precondition (can ride the #933 runner PR)
  • Record outcome in #928 once its results table exists (not yet created as of 2026-08-21)

Related

  • #929 — intp in comb_jit is intentional; documentation-only; this issue is the actual simplex_grid blocker
  • #942 — where the error surfaced; superseded by #929's keep-intp decision
  • #930 / PR #943 — PR #943 removes comb_jit's eager signature (and touches util/numba.py); per the Options table that changes when the crash appears but does not fix it, and the PR should not be read as a fix for this issue
  • #938 (merged) — ci/wasm/smoke_test.py with test_simplex_grid; #933 — the parked browser runner job where the warm-cache regression test and expected-failure list belong
  • #928 — Phase 0 results table (not yet created)
  • #927 — the other upstream-blocked Phase 1 item (jitted generators hang); neither this issue nor #927 should gate the intermediate release

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ci/wasm/smoke_test.py and read test_simplex_grid, then review emscripten-forge/recipes#6309 and the #933 browser runner context. Reproduce with a warm cache or pre-seeded /drive/.cache/numba. Done means the regression test covers the warm-cache case, is listed as an expected failure until the upstream fix ships, and its comment cites this issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, wasm
Domain
build-system, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.