es-ude / es-ude/OnDeviceTraining
arithmetic: evaluate third-party kernel libraries (CMSIS-NN/DSP, …) vs hand-rolled kernels
- Dominant language
- C
- Stars
- 1
- Forks
- 3
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 8
Description
## Question
Should the arithmetic layer adopt an existing optimized kernel library instead of hand-rolled matmul/conv kernels? Potential upside: large speedups (SIMD/DSP-extension-tuned kernels) and possibly memory reductions. Cost: an external dependency and a foreign quantization contract. This issue is the **investigation**, not an adoption decision.
## Candidates
- **CMSIS-DSP / CMSIS-NN** (Cortex-M; DSP/MVE-tuned fixed-point q7/q15/q31 kernels; Apache-2.0) — the primary candidate for our targets
- **muRISCV-NN** (RISC-V port of CMSIS-NN)
- TFLite-Micro's optimized kernel set (wraps CMSIS-NN on Arm; useful as a benchmark reference)
- Host-side BLAS family — only relevant for offline-sweep speed, not for MCU targets
- Eigen/uTensor-style C++ libraries are likely disqualified outright (pure-C codebase)
## Evaluation criteria
1. **Quantization-contract fit** — our SYM model (per-tensor float scale, int12 operand rule #227, hard int32-accumulator-only rule) vs. CMSIS-NN's fixed q-format and requant conventions. A mismatch here costs conversions that can eat the kernel gains.
2. **Training coverage** — these libraries target inference; backward/grad kernels would remain ours. The win may be forward-only; quantify what fraction of cycles that is (trace/profile an example, e.g. HAR).
3. **Integration seam** — the executeOp funnel with kernel function pointers (`opSpec_t.kernel`) is a natural backend boundary: a per-target adapter keeps the dependency optional (host = hand-rolled reference for bit-parity testing, MCU = library) without forking the framework.
4. Code size + scratch memory, license, maintenance cadence, toolchain fit (arm-gcc in devenv).
## Method
Benchmark-first: matmul + conv1d microbenchmarks, hand-rolled vs. candidate, on a representative Cortex-M and on host; plus the forward-vs-backward cycle split from (2). Deliverable = decision memo with numbers. If the answer is "adopt", implementation gets its own issue with the adapter design.
Contributor guide
Research direction
Start with the executeOp funnel and opSpec_t.kernel function-pointer seam, then review the existing hand-rolled matmul and conv1d kernels. Run the proposed microbenchmarks on a representative Cortex-M and host, and trace HAR to compare forward and backward cycles. Done means a decision memo with performance, conversion, code-size, scratch-memory, license, maintenance, and toolchain findings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- embedded-iot, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100