es-ude / es-ude/OnDeviceTraining

arithmetic: evaluate third-party kernel libraries (CMSIS-NN/DSP, …) vs hand-rolled kernels

Open
#336 0 comments 0 reactions 0 assignees View on GitHub
research
Dominant language
C
Stars
1
Forks
3
Avg merge
1d 1h
Merged PRs (30d)
8

Description

## Question

Should the arithmetic layer adopt an existing optimized kernel library instead of hand-rolled matmul/conv kernels? Potential upside: large speedups (SIMD/DSP-extension-tuned kernels) and possibly memory reductions. Cost: an external dependency and a foreign quantization contract. This issue is the **investigation**, not an adoption decision.

## Candidates

- **CMSIS-DSP / CMSIS-NN** (Cortex-M; DSP/MVE-tuned fixed-point q7/q15/q31 kernels; Apache-2.0) — the primary candidate for our targets
- **muRISCV-NN** (RISC-V port of CMSIS-NN)
- TFLite-Micro's optimized kernel set (wraps CMSIS-NN on Arm; useful as a benchmark reference)
- Host-side BLAS family — only relevant for offline-sweep speed, not for MCU targets
- Eigen/uTensor-style C++ libraries are likely disqualified outright (pure-C codebase)

## Evaluation criteria

1. **Quantization-contract fit** — our SYM model (per-tensor float scale, int12 operand rule #227, hard int32-accumulator-only rule) vs. CMSIS-NN's fixed q-format and requant conventions. A mismatch here costs conversions that can eat the kernel gains.
2. **Training coverage** — these libraries target inference; backward/grad kernels would remain ours. The win may be forward-only; quantify what fraction of cycles that is (trace/profile an example, e.g. HAR).
3. **Integration seam** — the executeOp funnel with kernel function pointers (`opSpec_t.kernel`) is a natural backend boundary: a per-target adapter keeps the dependency optional (host = hand-rolled reference for bit-parity testing, MCU = library) without forking the framework.
4. Code size + scratch memory, license, maintenance cadence, toolchain fit (arm-gcc in devenv).

## Method

Benchmark-first: matmul + conv1d microbenchmarks, hand-rolled vs. candidate, on a representative Cortex-M and on host; plus the forward-vs-backward cycle split from (2). Deliverable = decision memo with numbers. If the answer is "adopt", implementation gets its own issue with the adapter design.

Contributor guide

Open the contributing guide

Research direction

Start with the executeOp funnel and opSpec_t.kernel function-pointer seam, then review the existing hand-rolled matmul and conv1d kernels. Run the proposed microbenchmarks on a representative Cortex-M and host, and trace HAR to compare forward and backward cycles. Done means a decision memo with performance, conversion, code-size, scratch-memory, license, maintenance, and toolchain findings.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
embedded-iot, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.