emscripten-core / emscripten-core/emscripten

Decouple test suite from compiler internals for pure blackbox testing

Open
#27,666 1 comment 0 reactions 0 assignees View on GitHub
tests
Dominant language
C++
Stars
27.6k
Forks
3.6k
Avg merge
1d 1h
Merged PRs (30d)
105

Description

Most tests in `test/test_core.py` and `test/test_browser.py` test `emcc` as a black box via CLI invocations. However, parts of `test/test_other.py`, `test/test_sanity.py`, and the test harness (`test/common.py`) directly import and depend on Python compiler internals (`tools/*`).

If we want to rewrite the compiler (or components like the linker) in another language, or run the test suite against alternative implementations, we need to eliminate these internal dependencies and make the suite purely blackbox.

---

## Breakdown & Estimated Effort

| Area | Current Coupling | Proposed Fix | Effort |
| :--- | :--- | :--- | :--- |
| **1. Global State Mutation** | Tests for `-print-*` flags in `test_other.py` mutate `options.lto` and `settings.MEMORY64` to predict paths via `cache.get_lib_dir()`. | Assert on CLI output paths directly without altering Python singletons. | **Small** (~0.5 day) |
| **2. Binary Type Checks** | `test_other.py` calls `building.is_wasm()`, `is_wasm_dylib()`, `is_ar()`. | Inspect magic bytes (`b'\0asm'`) or use `llvm-readobj` / `llvm-ar`. | **Small** (~1 day) |
| **3. Direct Unit Tests** | `test_other.py` / `test_browser.py` unit-test Python functions directly (`binary_encode`, `wasm_sourcemap`, `response_file`, `closure_compiler`). | Test via CLI flags or isolate into a dedicated Python unit test directory. | **Small** (~1–2 days) |
| **4. Wasm Inspection** | ~15 tests in `test_core.py` and `test_other.py` use `tools.webassembly.Module` to check section names/features. | Replace with standard CLI tools (`llvm-readobj`, `llvm-objdump -h`, `wasm-dis`). | **Medium** (~2–3 days) |
| **5. Cache & Ports Introspection** | Tests query `cache.get_sysroot_dir()` or `ports.Ports.get_dir()` to find files in the cache. | Expose paths via CLI (e.g. `emcc --print-sysroot`) and test ports via `--use-port`. | **Medium** (~2 days) |
| **6. Harness Settings Filtering** | `common.py` imports `COMPILE_TIME_SETTINGS` to filter `-s` flags during compile-only (`-c`) runs. | Make compiler ignore/warn on link settings during `-c`, or query capabilities via CLI. | **Medium** (~2–3 days) |
| **7. Sanity Suite Partitioning** | `test_sanity.py` (~1,000 lines) tests the Python toolchain driver (`config.py`, cache locking, version checks). | Separate Python driver tests from the language-agnostic conformance test suite. | **Medium** (~1–2 days) |

**Total Estimated Effort:** **~2 to 3 weeks**

---

## Next Steps

1. Clean up quick wins in `test_other.py` (remove `options`/`settings` mutation, replace `building.is_*` helpers).
2. Replace `tools.webassembly` calls with standard binary tools (`llvm-readobj`).
3. Decouple harness dependencies (`COMPILE_TIME_SETTINGS`) and separate `test_sanity.py`.

Contributor guide

Open the contributing guide

Research direction

Start with the quick-win areas in test/test_other.py, then trace the direct imports in test/test_browser.py, test/test_sanity.py, and test/common.py. Run the affected black-box CLI tests and compare their use of tools/*, options/settings, cache, ports, and webassembly helpers. Done means the listed suites exercise compiler behavior through CLI or explicitly isolated tests without those internal dependencies.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, wasm
Domain
compilers, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.