emscripten-core / emscripten-core/emscripten
Decouple test suite from compiler internals for pure blackbox testing
- Dominant language
- C++
- Stars
- 27.6k
- Forks
- 3.6k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 105
Description
Most tests in `test/test_core.py` and `test/test_browser.py` test `emcc` as a black box via CLI invocations. However, parts of `test/test_other.py`, `test/test_sanity.py`, and the test harness (`test/common.py`) directly import and depend on Python compiler internals (`tools/*`).
If we want to rewrite the compiler (or components like the linker) in another language, or run the test suite against alternative implementations, we need to eliminate these internal dependencies and make the suite purely blackbox.
---
## Breakdown & Estimated Effort
| Area | Current Coupling | Proposed Fix | Effort |
| :--- | :--- | :--- | :--- |
| **1. Global State Mutation** | Tests for `-print-*` flags in `test_other.py` mutate `options.lto` and `settings.MEMORY64` to predict paths via `cache.get_lib_dir()`. | Assert on CLI output paths directly without altering Python singletons. | **Small** (~0.5 day) |
| **2. Binary Type Checks** | `test_other.py` calls `building.is_wasm()`, `is_wasm_dylib()`, `is_ar()`. | Inspect magic bytes (`b'\0asm'`) or use `llvm-readobj` / `llvm-ar`. | **Small** (~1 day) |
| **3. Direct Unit Tests** | `test_other.py` / `test_browser.py` unit-test Python functions directly (`binary_encode`, `wasm_sourcemap`, `response_file`, `closure_compiler`). | Test via CLI flags or isolate into a dedicated Python unit test directory. | **Small** (~1–2 days) |
| **4. Wasm Inspection** | ~15 tests in `test_core.py` and `test_other.py` use `tools.webassembly.Module` to check section names/features. | Replace with standard CLI tools (`llvm-readobj`, `llvm-objdump -h`, `wasm-dis`). | **Medium** (~2–3 days) |
| **5. Cache & Ports Introspection** | Tests query `cache.get_sysroot_dir()` or `ports.Ports.get_dir()` to find files in the cache. | Expose paths via CLI (e.g. `emcc --print-sysroot`) and test ports via `--use-port`. | **Medium** (~2 days) |
| **6. Harness Settings Filtering** | `common.py` imports `COMPILE_TIME_SETTINGS` to filter `-s` flags during compile-only (`-c`) runs. | Make compiler ignore/warn on link settings during `-c`, or query capabilities via CLI. | **Medium** (~2–3 days) |
| **7. Sanity Suite Partitioning** | `test_sanity.py` (~1,000 lines) tests the Python toolchain driver (`config.py`, cache locking, version checks). | Separate Python driver tests from the language-agnostic conformance test suite. | **Medium** (~1–2 days) |
**Total Estimated Effort:** **~2 to 3 weeks**
---
## Next Steps
1. Clean up quick wins in `test_other.py` (remove `options`/`settings` mutation, replace `building.is_*` helpers).
2. Replace `tools.webassembly` calls with standard binary tools (`llvm-readobj`).
3. Decouple harness dependencies (`COMPILE_TIME_SETTINGS`) and separate `test_sanity.py`.
Contributor guide
Research direction
Start with the quick-win areas in test/test_other.py, then trace the direct imports in test/test_browser.py, test/test_sanity.py, and test/common.py. Run the affected black-box CLI tests and compare their use of tools/*, options/settings, cache, ports, and webassembly helpers. Done means the listed suites exercise compiler behavior through CLI or explicitly isolated tests without those internal dependencies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, wasm
- Domain
- compilers, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100