es-ude / es-ude/OnDeviceTraining

model: small time-series transformer architectures

Open
#333 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C
Stars
1
Forks
3
Avg merge
1d 1h
Merged PRs (30d)
8

Description

## Feature

Small time-series transformer architectures (encoder-only, few blocks, small d_model) as a supported model family — attention-based sequence models over the same `[C, L]` data the conv examples consume.

## Gap analysis

**Have:** Linear, LayerNorm, Softmax, Dropout, Matmul arithmetic, zero-copy transpose (`orderOfDimensions`), CrossEntropy/MSE heads.

**Missing:**
- **Residual adds** around attention and FFN sub-blocks — blocked by **#329** (same Add-merge as ResNet skips).
- **Scaled dot-product attention** — QKᵀ batched over heads, 1/√d scaling, row-wise softmax, and the attention backward. Design decision: compose from existing ops through the #329 graph vs. a fused attention layer. Memory argues for fused (avoids materializing the L×L score tensor more often than necessary — on MCU the score matrix is the footprint driver).
- **Positional encoding** — sinusoidal constant-tensor add (cheap, no params) or learned parameter add.
- **Input embedding** — for time series this is just a Linear projection of each timestep; no token-embedding table needed.
- **GELU** — optional; a ReLU-FFN is acceptable for small models (documented deviation from the standard recipe).
- **Adam(W) #328** — soft prerequisite; transformers are practically untrainable with plain SGD at these scales.

## Quantization angle

FLOAT32 end-to-end first. Integer-only attention (softmax, 1/√d) is genuinely open research — explicitly out of scope here; later candidate for the integer-only-training track (#308/#309).

## Out of scope

Decoder/cross-attention, KV-caching, tokenizers — this is about small encoder-style TS models, not language models.

Contributor guide

Open the contributing guide

Research direction

Start by reading the residual-add prerequisite in #329 and the Adam(W) prerequisite in #328, then locate the existing Linear, LayerNorm, Softmax, Dropout, Matmul, transpose, and loss implementations. Done means a supported float32 encoder-only time-series transformer over [C, L], with a documented attention design and the stated decoder, quantization, and tokenizer exclusions.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
embedded-iot, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.