CaltechExperimentalGravity / CaltechExperimentalGravity/system_ident

CDS transport fault taxonomy, chunked live read with adjacency verification, and an asymmetric retry policy

Open
#35 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
0
Forks
1
PR merge metrics
No merged PRs in 30d

Description

**Spin-out of #32 · Stage C (#11, #12) · spec §4.3.1, §4.3.2, §4.3.3, §4.3.4, §4.3.7.**

Covers #32 items **1, 3, 8, 10**.

**Only one thing in this issue has been observed** (see "the weak empirical driver" below). The rest
is architecture for the multi-user, unattended case. Filed now because this is the single most
expensive thing to retrofit in the whole campaign.

## 1. The fault taxonomy — the highest-value item

`cds_transport.py` (#11) must define one exception hierarchy, and **both** `AWGNDSTransport` and
`TwinTransport` must raise the same types, so every handler downstream is testable on the #5 fakes:

```
CDSTransportError (base — catching this alone is enough to stop driving)
TransportUnavailable network error, framebuilder gone #32 items 3, 10
TestpointLost slot cleared by another user/process #32 items 1, 2
TestpointTimeout allocation did not complete #32 item 8
ChannelNotFound pre-flight existence #32 item 7
ChannelNotInjectable slow read-only EPICS as an excitation ch. #32 item 6
DataIntegrityError non-finite / short / drive mismatch #32 item 5
TimingFault start_time, sample count, rate, GPS #32 item 9
```

If the transports raise bare `RuntimeError`, every handler added later is a retrofit across the whole
backend. Two standing rules: an **unclassified** fault still hits the base and therefore still
triggers safe teardown — the default is safe; and a fault with no matching class is **a gap in the
taxonomy, not a licence for an ad-hoc handler** — add the class.

**The taxonomy is designed to absorb faults not listed in #32.** Place a new one by answering two
questions: (i) can we get data at all? no → *transport*; data arrives but is wrong → *integrity*; data
is fine but the plant never received the drive → *hardware-state*. (ii) is the channel recorded or a
test point? That decides whether a retry is even meaningful. Worked examples in spec §4.3.4 — a
front-end model restart, an excitation slot taken by another client, ADC/DAC overflow, framebuilder
disk full, a duotone discrepancy, an operator changing a filter-module gain mid-record — all land on
an existing class with no new mechanism.

## 2. `stream` is the primitive; `fetch` is derived

- [ ] `stream(channels, duration, chunk_s) -> Iterator[Capture]`, with `fetch` consuming it.
- [ ] Blocks are **verified adjacent** — block *k+1* begins exactly where *k* ended — and concatenated.
- [ ] **The request stays open for the whole record.** Per spec §4.3.1, many channels are not recorded
to disk and must be captured live or they are unretrievable; N back-to-back `getdata` calls
would lose samples at **every** boundary. This is a correctness requirement, not a style choice.

### Why chunk at all — strongest reason first

1. **Fault detection during the record.** A single blocking fetch discovers a framebuilder reboot, a
lost test point or a stream underrun *hours* late — spec §4.2's campaign estimates run to 2.5 h —
and leaves the #16 SIGINT handler nowhere to land.
2. **Bounded framebuilder resource use.** The binding constraint is *the resources available to the
framebuilder*: hardware with less memory, **and/or** a machine also running other tasks so memory
is shared. Either, or both.
3. **A weak empirical driver, recorded as such.** Framebuilders failing to provide long data stretches
**has actually been observed** — the one #32 item that is not hypothetical. The cause was **never
investigated** (candidates: a test-point timeout, framebuilder resource limits, a client-side
limitation) and workarounds are commonly used instead. **Not authoritative:** no numbers, no logs,
no root cause. It supports the two architectural reasons; it is not their basis.

### Concatenation is not stitching

| operation | rule |
|---|---|
| **concatenation** of verified-adjacent blocks | **required** — this *is* the continuous record, and it is invisible downstream |
| **fabrication** — zero-fill, interpolate, resample or realign across a hole | **forbidden always**: synthetic samples in X or Y is §2.1's failure class |
| **gap-tolerant reassembly** — analysing two contiguous runs separated by a known GPS gap | **disallowed** |

This does not weaken §2.1. That section forbids *correcting* the buffer; adjacency checking only
*verifies* it. Gap-tolerant reassembly is recorded as considered-and-rejected in spec §4.3.2 so it is
not re-derived: a common phase shift does cancel in the ratio-of-averages, but the period *straddling*
the seam carries a discontinuity that appears differently in X (a raw drive jump) than in Y (that jump
filtered by the plant), so `Y_seam ≠ H·X_seam`. **Disallowed as policy.**

## 3. Retry policy — asymmetric on two axes

- [ ] **Passive reads retry** — bounded, with backoff, for `TransportUnavailable` / `TestpointTimeout`.
A passive record may also be re-attempted after a gap: it carries no excitation.
- [ ] **Excited records never retry automatically.** Re-taking one means re-injecting, which is a new
actuation needing a new approval token under Rule 2 (#17).
- [ ] A retry never silently extends a record's duration or the campaign's time budget.

## 4. What `TwinTransport` must not pretend (#12)

It must raise every type and emit chunked blocks so all of the above is testable off-hardware. But
respect the real hardware/simulation difference rather than papering over it: in the twin, GPS is
**instantiated at runtime** and each `mdl.run()` is a fresh evolution, so continuity *across* records
is meaningless there and re-acquiring a whole stretch is cheap. Neither holds on hardware, where GPS
comes from an antenna (best case) or an NTP server (worst case). A gap *within* one twin record **is**
representable — keep running cycles, do not fetch them — and that is what the gap tests should use.

Note also that an NTP-disciplined clock can step **backwards**, so GPS monotonicity stays a check and
a backwards step is a `TimingFault`.

## Config surface

Minimal, honouring #31's "no new configuration" instinct: only `cds.read_chunk_s` (validated
`<= segment_duration`) and `cds.passive_read_retries`, both with in-code defaults. **The chunk-size
default is provisional** — per the feasibility gate, no value can be justified without a framebuilder
measurement, and the number belongs to that machine's available resources, not to us.

---
**Campaign:** CDS hardware backend · branch [`feat/cds-hardware-backend`](https://github.com/CaltechExperimentalGravity/system_ident/tree/feat/cds-hardware-backend)
· [spec](https://github.com/CaltechExperimentalGravity/system_ident/blob/feat/cds-hardware-backend/docs/superpowers/specs/2026-08-03-cds-hardware-backend-design.md) · [plan](https://github.com/CaltechExperimentalGravity/system_ident/blob/feat/cds-hardware-backend/docs/superpowers/plans/2026-08-03-cds-hardware-backend.md) · [handoff](https://github.com/CaltechExperimentalGravity/system_ident/blob/feat/cds-hardware-backend/notes/cds-hardware-bringup-2026-08.md)
*Code is deferred until the plan and issues have been reviewed.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Review cds_transport.py, the #5 fakes, and spec §4.3 alongside the linked plan before choosing an implementation slice. Done means both transports share the listed fault hierarchy, stream emits verified-adjacent chunks, passive retries are bounded while excited records do not retry, and the two cds settings are validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.