CaltechExperimentalGravity / CaltechExperimentalGravity/system_ident
CDS transport fault taxonomy, chunked live read with adjacency verification, and an asymmetric retry policy
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
**Spin-out of #32 · Stage C (#11, #12) · spec §4.3.1, §4.3.2, §4.3.3, §4.3.4, §4.3.7.**
Covers #32 items **1, 3, 8, 10**.
**Only one thing in this issue has been observed** (see "the weak empirical driver" below). The rest
is architecture for the multi-user, unattended case. Filed now because this is the single most
expensive thing to retrofit in the whole campaign.
## 1. The fault taxonomy — the highest-value item
`cds_transport.py` (#11) must define one exception hierarchy, and **both** `AWGNDSTransport` and
`TwinTransport` must raise the same types, so every handler downstream is testable on the #5 fakes:
```
CDSTransportError (base — catching this alone is enough to stop driving)
TransportUnavailable network error, framebuilder gone #32 items 3, 10
TestpointLost slot cleared by another user/process #32 items 1, 2
TestpointTimeout allocation did not complete #32 item 8
ChannelNotFound pre-flight existence #32 item 7
ChannelNotInjectable slow read-only EPICS as an excitation ch. #32 item 6
DataIntegrityError non-finite / short / drive mismatch #32 item 5
TimingFault start_time, sample count, rate, GPS #32 item 9
```
If the transports raise bare `RuntimeError`, every handler added later is a retrofit across the whole
backend. Two standing rules: an **unclassified** fault still hits the base and therefore still
triggers safe teardown — the default is safe; and a fault with no matching class is **a gap in the
taxonomy, not a licence for an ad-hoc handler** — add the class.
**The taxonomy is designed to absorb faults not listed in #32.** Place a new one by answering two
questions: (i) can we get data at all? no → *transport*; data arrives but is wrong → *integrity*; data
is fine but the plant never received the drive → *hardware-state*. (ii) is the channel recorded or a
test point? That decides whether a retry is even meaningful. Worked examples in spec §4.3.4 — a
front-end model restart, an excitation slot taken by another client, ADC/DAC overflow, framebuilder
disk full, a duotone discrepancy, an operator changing a filter-module gain mid-record — all land on
an existing class with no new mechanism.
## 2. `stream` is the primitive; `fetch` is derived
- [ ] `stream(channels, duration, chunk_s) -> Iterator[Capture]`, with `fetch` consuming it.
- [ ] Blocks are **verified adjacent** — block *k+1* begins exactly where *k* ended — and concatenated.
- [ ] **The request stays open for the whole record.** Per spec §4.3.1, many channels are not recorded
to disk and must be captured live or they are unretrievable; N back-to-back `getdata` calls
would lose samples at **every** boundary. This is a correctness requirement, not a style choice.
### Why chunk at all — strongest reason first
1. **Fault detection during the record.** A single blocking fetch discovers a framebuilder reboot, a
lost test point or a stream underrun *hours* late — spec §4.2's campaign estimates run to 2.5 h —
and leaves the #16 SIGINT handler nowhere to land.
2. **Bounded framebuilder resource use.** The binding constraint is *the resources available to the
framebuilder*: hardware with less memory, **and/or** a machine also running other tasks so memory
is shared. Either, or both.
3. **A weak empirical driver, recorded as such.** Framebuilders failing to provide long data stretches
**has actually been observed** — the one #32 item that is not hypothetical. The cause was **never
investigated** (candidates: a test-point timeout, framebuilder resource limits, a client-side
limitation) and workarounds are commonly used instead. **Not authoritative:** no numbers, no logs,
no root cause. It supports the two architectural reasons; it is not their basis.
### Concatenation is not stitching
| operation | rule |
|---|---|
| **concatenation** of verified-adjacent blocks | **required** — this *is* the continuous record, and it is invisible downstream |
| **fabrication** — zero-fill, interpolate, resample or realign across a hole | **forbidden always**: synthetic samples in X or Y is §2.1's failure class |
| **gap-tolerant reassembly** — analysing two contiguous runs separated by a known GPS gap | **disallowed** |
This does not weaken §2.1. That section forbids *correcting* the buffer; adjacency checking only
*verifies* it. Gap-tolerant reassembly is recorded as considered-and-rejected in spec §4.3.2 so it is
not re-derived: a common phase shift does cancel in the ratio-of-averages, but the period *straddling*
the seam carries a discontinuity that appears differently in X (a raw drive jump) than in Y (that jump
filtered by the plant), so `Y_seam ≠ H·X_seam`. **Disallowed as policy.**
## 3. Retry policy — asymmetric on two axes
- [ ] **Passive reads retry** — bounded, with backoff, for `TransportUnavailable` / `TestpointTimeout`.
A passive record may also be re-attempted after a gap: it carries no excitation.
- [ ] **Excited records never retry automatically.** Re-taking one means re-injecting, which is a new
actuation needing a new approval token under Rule 2 (#17).
- [ ] A retry never silently extends a record's duration or the campaign's time budget.
## 4. What `TwinTransport` must not pretend (#12)
It must raise every type and emit chunked blocks so all of the above is testable off-hardware. But
respect the real hardware/simulation difference rather than papering over it: in the twin, GPS is
**instantiated at runtime** and each `mdl.run()` is a fresh evolution, so continuity *across* records
is meaningless there and re-acquiring a whole stretch is cheap. Neither holds on hardware, where GPS
comes from an antenna (best case) or an NTP server (worst case). A gap *within* one twin record **is**
representable — keep running cycles, do not fetch them — and that is what the gap tests should use.
Note also that an NTP-disciplined clock can step **backwards**, so GPS monotonicity stays a check and
a backwards step is a `TimingFault`.
## Config surface
Minimal, honouring #31's "no new configuration" instinct: only `cds.read_chunk_s` (validated
`<= segment_duration`) and `cds.passive_read_retries`, both with in-code defaults. **The chunk-size
default is provisional** — per the feasibility gate, no value can be justified without a framebuilder
measurement, and the number belongs to that machine's available resources, not to us.
---
**Campaign:** CDS hardware backend · branch [`feat/cds-hardware-backend`](https://github.com/CaltechExperimentalGravity/system_ident/tree/feat/cds-hardware-backend)
· [spec](https://github.com/CaltechExperimentalGravity/system_ident/blob/feat/cds-hardware-backend/docs/superpowers/specs/2026-08-03-cds-hardware-backend-design.md) · [plan](https://github.com/CaltechExperimentalGravity/system_ident/blob/feat/cds-hardware-backend/docs/superpowers/plans/2026-08-03-cds-hardware-backend.md) · [handoff](https://github.com/CaltechExperimentalGravity/system_ident/blob/feat/cds-hardware-backend/notes/cds-hardware-bringup-2026-08.md)
*Code is deferred until the plan and issues have been reviewed.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Review cds_transport.py, the #5 fakes, and spec §4.3 alongside the linked plan before choosing an implementation slice. Done means both transports share the listed fault hierarchy, stream emits verified-adjacent chunks, passive retries are bounded while excited records do not retry, and the two cds settings are validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100