altunenes / altunenes/parakeet-rs
Tracking burn onnx for a possible pure Rust future
- Langage dominant
- Rust
- Étoiles
- 393
- Forks
- 59
- Merge moyen
- 3 h 59 min
- PR mergées (30 j)
- 8
Description
opening this just to make people aware of something I've been watching.
tracel-ai released burn-onnx 0.21 recently as a dedicated repo, split off from the main burn crate. it converts onnx → rust source code at build time and ships with burn as the runtime (cpu via burn-flex, gpu via wgpu / metal / cuda etc).
I think the last time I tried Burn ONNX was about two years ago, and to be honest, I didn’t get very far because it didn’t support many ONNX operators. However, from what I’ve seen, there’s been a lot of serious work done recently, and it looks like more is on the way (and at a rapid pace).
so that means no onnxruntime dylib, no microsoft c++ runtime, no separate binary lobs to bundle. for cross-platform via wgpu especially this would be a massive win currently for instance webgpu via ort is still beta-ish and coreml falls back to cpu half the time on these models, so a pure rust path would solve a lot.
but I'm not migrating right now and I want to be upfront about why.
current burn-onnx is build-time only. you give it an onnx in build.rs, it generates rust source code that gets baked into your binary at cargo build. that's fine for an app shipping one fixed model. parakeet-rs is a library where users do from_pretrained("./model_path", None) at runtime and pass whatever model directory they downloaded. with build-time only, every model swap = full recompile of the library. and, to be honest, that’s not exactly the scenario we’re hoping for in this production...
second blocker: in burn the backend is currently a compile time generic. our public types would become Parakeet, ParakeetTDT etc, which is viral and pretty rough on user code. or we lock one backend and lose the choice. burn 0.22 reportedly fixes this.
so for now I will:
- watch burn 0.22 (backend flexibility lands there)
- watch the runtime-import direction in burn-onnx
- when both ship and stabilize, do a tiny proof of concept with CTC or TDT on wgpu, see if codegen + numerics hold upon a real asr model.
Their Kokoro TTS model check is actually really encouraging btw when I inspect. they hit f32 precision issues in matrix-DFT/STFT and fixed it with internal f64 matmul. exactly the kind of stuff we'd hit too, and they're being honest about the residual divergence rather than hiding it.
one more thing worth flagging: even after both blockers land, compile times are real. kokoro v1.0 takes ~7 min to codegen+compile on an m3 per their docs. parakeet-rs ships ~6 models, that's potentially serious ci time. not a dealbreaker but not free either.
so it's a watchlist for now. if anyone here has tried burn-onnx on asr-like graphs or got the wgpu backend running on linux/windows, i'd love to hear about it. or if anyone has stronger opinions than mine about whether the pure Rust path is worth the eventual transition cost also welcome. However, I want to point out that this will happen. The only question is when?
some links:
- repo: https://github.com/tracel-ai/burn-onnx
- the kokoro model check (worth reading for the real-world precision story): https://github.com/tracel-ai/burn-onnx/tree/main/crates/model-checks/kokoro
- 0.21 release notes: https://github.com/tracel-ai/burn-onnx/releases/tag/v0.21.0
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.