DiamondLightSource / DiamondLightSource/smartem-decisions
Acquisitions never advance past status='started' - no reliable end-detection signal
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Symptom
All 137 acquisitions in the current dev DB have `status = 'started'` and `end_time = null`. The status flag is never advanced to `completed`, `paused`, or `abandoned` regardless of how long ago the acquisition began (some have `start_time` from Dec 2025).
The smartem-frontend dashboard surfaced this while wiring the Active/Recent partition — every record landed in Active. A UI heuristic (24h recency window) was briefly tried and reverted: cryo-EM acquisitions legitimately run well past 24 hours, so the UI cannot infer completion from age. This belongs in the backend (or agent), not the frontend.
## Where the gap is
Two possible places, both probably needed:
1. **Agent side** — the agent currently watches the EPU output directory and POSTs acquisitions/grids/foilholes/etc. as they appear, but has no rule for "this acquisition is now over." Candidates the agent could observe:
- EPU process exit / session JSON closing tag / specific terminal log line
- A configurable idle-timeout: no new files in the acquisition directory for N hours → mark complete
- An explicit user gesture (out of scope without UI work)
2. **Backend side** — the API accepts the update but currently has no rule that demotes `started` to `completed` autonomously. If we go the idle-timeout route, the backend could run a periodic sweep instead of (or in addition to) the agent doing it locally.
A combination is likely: the agent emits the most reliable signals it can (process exit, session close), the backend runs an idle-timeout sweep as a safety net for sessions where the agent died or restarted mid-acquisition.
## Impact
- Dashboard activity partition is meaningless until this lands — the smartem-frontend currently shows the entire history as Active.
- Any downstream consumer (ARIA depositions, dashboards, archival policies) that gates on \"is this acquisition done\" has no signal to gate on.
- Grid/foilhole/micrograph statuses *do* advance (we have `motion correction completed`, `ctf completed`, etc.), so the gap is specifically at the acquisition envelope level.
## Acceptance criteria
- An acquisition with no recent filesystem activity (or whose EPU process is known to have exited) gets `status` set to `completed` (or `abandoned`, depending on whether grids were collected) and `end_time` set to the best-available timestamp.
- The mechanism is documented (agent README or an ADR) so the heuristic isn't surprising.
- Status transitions are tested.
## Related
- smartem-frontend PR #107 wires the dashboard to real API; the comment thread there has the Playwright walkthrough that surfaced this.
- smartem-devtools#81 is about exposing a microscope list (orthogonal, but mentioned here because it's also blocking dashboard completeness).
Contributor guide
Assessment
This issue has not been assessed yet.