ConduitIO / ConduitIO/conduit

Standalone processors are recompiled from scratch on every boot (~4s each, no compilation cache)

Open
#2,862 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
610
Forks
63
Avg merge
12h 28m
Merged PRs (30d)
57

Description

## Summary

`conduit run` recompiles **every** `.wasm` under `--processors.path` on **every boot**. There is no wazero compilation cache anywhere in the repo — `grep -rn "CompilationCache"` returns nothing outside design text. With wazevo (the default on amd64/arm64) a 19 MB module costs ~3.8s, so a user with five installed processors pays ~19s of pure recompilation on every start, and it recurs on every restart, every crash-loop iteration, and every container start.

Found while investigating `make test` runtime (#2861); the test-side symptom is fixed there, this is the production half and it is the more valuable one.

## Path

```
pkg/conduit/runtime.go:519 proc_standalone.NewRegistry
-> standalone/registry.go:115 reloadPlugins
-> loadPlugins: iterate every non-directory entry under --processors.path
-> standalone/registry.go:273 r.runtime.CompileModule(ctx, wasmBytes)
```

`newRuntime` is `wazero.NewRuntime` with the default `RuntimeConfig` — no `WithCompilationCache`. The default processors path is `/processors` (`pkg/conduit/config.go:327`), which is exactly where `conduit processor-plugins install` writes artifacts, so this hits anyone using standalone processors at all.

## Measurements

Independently reproduced by two sessions, against the real 18.4 MB chaos fixture, no `-race`, M-series arm64:

| configuration | run 1 | run 2 | run 3 |
| --- | --- | --- | --- |
| compiler, no cache (**what ships today**) | 3.811s | 3.810s | — |
| interpreter | 0.265s | 0.264s | — |
| compiler + in-memory `wazero.NewCompilationCache()` | 3.736s | 3.811s | 3.801s |
| compiler + `NewCompilationCacheWithDir` | 3.793s | **0.111s** | **0.112s** |

Two non-obvious results worth recording:

- **An in-memory `CompilationCache` gives zero reuse.** `CompiledModule.Close()` evicts the entry, so the obvious fix does nothing. Anyone attempting this should not stop when the in-memory version fails to help.
- **Only `NewCompilationCacheWithDir` survives** module close and process exit — 34x on compile alone, ~50x measured across the full boot path.

## Why this needs a design doc rather than a patch

It introduces on-disk state Conduit does not have today, so at minimum:

- **Location.** A new directory, or under the existing state dir? Interaction with `--processors.path`, containers, and read-only root filesystems.
- **Invalidation.** Keyed on module bytes, wazero version, GOARCH, and Conduit version. A stale entry after a wazero bump would execute code compiled by a different compiler version — this is the part that needs care.
- **Config surface.** On by default? A `--processors.compilation-cache.*` family? An opt-out for reproducibility-sensitive deployments.
- **Permissions and multi-tenancy.** A shared cache directory is a code-execution surface; two Conduit instances as different users must not be able to poison each other.
- **Failure mode.** A corrupt or unwritable cache must degrade to today's behavior, never fail startup.

## Architecture note

This only bites where wazevo is active. `NewRuntimeConfig()` is `engineKindAuto` and selects the compiler only when `platform.CompilerSupported()` — amd64/arm64 on the mainstream OSes. On `linux/386` (which this repo has a required build check for), production already runs the interpreter at ~0.26s and there is nothing to cache.

## Suggested acceptance

- Second and subsequent boots with unchanged processor binaries skip compilation, demonstrated with a timing test.
- A wazero version bump, a Conduit version bump, or a changed module invalidates the entry.
- An unwritable or corrupt cache directory logs and falls back to compiling, never fails startup.

Contributor guide

Open the contributing guide

Research direction

Start with pkg/conduit/runtime.go at proc_standalone.NewRegistry and follow standalone/registry.go reloadPlugins/loadPlugins to the CompileModule call; also review the default processors path in pkg/conduit/config.go. Define the cache design and validation behavior, then demonstrate that unchanged processor binaries avoid compilation while version or module changes invalidate entries and cache failures fall back without blocking startup.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, wasm
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.