es-ude / es-ude/OnDeviceTraining
model: small time-series transformer architectures
- Dominant language
- C
- Stars
- 1
- Forks
- 3
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 8
Description
## Feature
Small time-series transformer architectures (encoder-only, few blocks, small d_model) as a supported model family — attention-based sequence models over the same `[C, L]` data the conv examples consume.
## Gap analysis
**Have:** Linear, LayerNorm, Softmax, Dropout, Matmul arithmetic, zero-copy transpose (`orderOfDimensions`), CrossEntropy/MSE heads.
**Missing:**
- **Residual adds** around attention and FFN sub-blocks — blocked by **#329** (same Add-merge as ResNet skips).
- **Scaled dot-product attention** — QKᵀ batched over heads, 1/√d scaling, row-wise softmax, and the attention backward. Design decision: compose from existing ops through the #329 graph vs. a fused attention layer. Memory argues for fused (avoids materializing the L×L score tensor more often than necessary — on MCU the score matrix is the footprint driver).
- **Positional encoding** — sinusoidal constant-tensor add (cheap, no params) or learned parameter add.
- **Input embedding** — for time series this is just a Linear projection of each timestep; no token-embedding table needed.
- **GELU** — optional; a ReLU-FFN is acceptable for small models (documented deviation from the standard recipe).
- **Adam(W) #328** — soft prerequisite; transformers are practically untrainable with plain SGD at these scales.
## Quantization angle
FLOAT32 end-to-end first. Integer-only attention (softmax, 1/√d) is genuinely open research — explicitly out of scope here; later candidate for the integer-only-training track (#308/#309).
## Out of scope
Decoder/cross-attention, KV-caching, tokenizers — this is about small encoder-style TS models, not language models.
Contributor guide
Research direction
Start by reading the residual-add prerequisite in #329 and the Adam(W) prerequisite in #328, then locate the existing Linear, LayerNorm, Softmax, Dropout, Matmul, transpose, and loss implementations. Done means a supported float32 encoder-only time-series transformer over [C, L], with a documented attention design and the stated decoder, quantization, and tokenizer exclusions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- embedded-iot, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100