Spike: .NET sidecar deserialiser for EPU .dm/.xml (Option C) using NetDataContractSerializer

## Summary

Spike a **.NET sidecar deserialiser** for EPU `.dm`/`.xml` files that loads EPU's own
assemblies and uses `NetDataContractSerializer` to rehydrate the original object graph,
emitting clean JSON to the Python agent - instead of hand-rolling `lxml` XPath against the
serialiser's internal structures.

This is **Option C** from the investigation of the EPU serialisation format. The goal of this
issue is a proof-of-concept plus a go/no-go decision, **not** a committed rewrite.

## Background

EPU (ThermoFisher, closed-source, Windows .NET Framework 4.x) does not write a bespoke
interchange schema. The `.dm` files are `NetDataContractSerializer` dumps of EPU's in-memory
CLR object graph, and the per-image `.xml` sidecars are `DataContractSerializer` output. Full
write-up: `smartem-devtools/docs/decision-records/epu-data-structures.md` (section "File
Serialisation Format").

Because it is a runtime object-graph dump, the payload is entangled with serialiser plumbing:

- Data-bearing collections are serialised **only** in their `...Efficient` (`ConcurrentDictionary`)
form; the plain domain lists are `i:nil`. Verified: a 1.8 MB `GridSquare_*.dm` with 534 foil
holes has a nil `TargetLocations` list and all 534 entries under
`TargetLocationsEfficient/m_serializationArray`. Atlas tiles behave the same via
`TilesEfficient/_items`.
- Per-entry element names carry an unstable generated type-hash suffix
(`KeyValuePairOfintTargetLocationXml...`).

The current agent parser (`src/smartem_agent/fs_parser.py`, e.g. the foil-hole read at
`TargetLocationsEfficient/m_serializationArray`, and the atlas-tile read at `TilesEfficient/_items`)
handles this correctly today with namespace-aware XPath plus `local-name()` prefix matching, so
**this is not a bug fix**. The question this spike answers is whether delegating deserialisation
to the vendor's own code is a worthwhile robustness/maintainability improvement.

## Proposal (Option C)

A small **.NET Framework 4.x** console helper, run on the EPU workstation next to the agent:

1. Loads EPU's own assemblies present on the workstation
(`Applications.Epu.Persistence.dll`, `Applications.SciencesAppsShared.*.dll`, `Fei.*.dll`).
2. Calls `NetDataContractSerializer.ReadObject()` on a `.dm` file to rehydrate the exact
`EpuSessionXml` / `GridSquareXml` / `AtlasSessionXml` / `FoilHoleCollectionXml` objects.
3. Walks the strongly-typed graph and serialises the domain fields we care about to a stable,
flat **JSON contract** on stdout.
4. The Python agent invokes it per file (subprocess) and maps the JSON to the existing
`smartem_common.schemas` pydantic models - the `fs_watcher`/orphan/queue machinery is unchanged.

The vendor's code owns refs, cycles, polymorphism and the `Efficient` dictionaries, so our side
never touches serialiser internals.

## Explicit scope - what this does and does NOT solve

Removes:
- Coupling to serialiser plumbing (`z:*` object-graph attributes, `m_serialization*` bookkeeping,
the `KeyValuePairOfint...` type-hash suffix, the `TargetLocationsEfficient`/`TilesEfficient`
shapes).

Does **not** remove (out of scope - do not conflate):
- **Filesystem sourcing.** It still reads the `.dm` files; it does not consume from an upstream API.
- **Liveness via fs watch.** We still need `watchdog` events to know *when* to (re)read a file.
- **Domain-schema coupling.** If Thermo renames `GridSquareXml.State`, we still break - same as now.

If the objective is to escape the filesystem entirely, that is a different track (Smart EPU Open
API / Athena-DMP) and should be a separate issue - see Alternatives.

## Dev-ready plan

**Phase 0 - feasibility spike (timeboxed, ~1-2 days)**
- [ ] Locate the EPU install on a DLS EPU workstation; confirm the persistence assemblies are
present and loadable, and record their versions (dataset under test was built by EPU
`3.9.1.8206`; assemblies stamp their own versions).
- [ ] Minimal C# program: load `Applications.Epu.Persistence`, `NetDataContractSerializer.ReadObject()`
one real `GridSquare_*.dm` (use the large 534-foil-hole file), dump the foil-hole count and a
couple of fields. Prove the graph rehydrates without our own assemblies.
- [ ] Decide runtime: native .NET Framework 4.x console (invoked as subprocess) vs `pythonnet`
in-process. Default assumption: **subprocess emitting JSON** (clean process boundary, no
interop pinning).

**Phase 1 - JSON contract + parity**
- [ ] Define the stdout JSON schema (mirror `smartem_common.schemas`: `AcquisitionData`,
`AtlasData`+tiles+gridsquare_positions, `GridSquareMetadata`+foilhole_positions,
`FoilHoleData`, `MicrographManifest`, `MicroscopeData`).
- [ ] Implement the graph walk for all root types: `EpuSessionXml`, `AtlasSessionXml`,
`SampleXml`, `ScreeningSessionXml`, `GridSquareXml` (+ nested `FoilHoleCollectionXml`), and
the `MicroscopeImage` sidecars.
- [ ] **Parity harness**: run both paths (current `fs_parser` and the .NET helper) over the
`bi37708-28` dataset and diff the resulting pydantic models. Zero diffs = pass.

**Phase 2 - integration + decision gate**
- [ ] Wire the helper behind a config flag in the agent intake path (fallback to `fs_parser`).
- [ ] Package the helper for the Windows agent build (`.exe`); confirm it runs where the agent runs.
- [ ] Write up findings and a recommendation on the ADR track; **go/no-go** on adopting it.

## Acceptance criteria (for the spike)

- A working .NET helper rehydrates a real `GridSquare_*.dm` via `NetDataContractSerializer` using
EPU's own assemblies, with no reimplementation of their types on our side.
- Byte-for-byte (or field-for-field) parity with the current parser across the `bi37708-28`
dataset, demonstrated by the parity harness.
- A written recommendation covering the risks below, ending in go/no-go.

## Risks / open questions

- **`NetDataContractSerializer` is not in .NET Core/.NET 5+** (Framework-only; a `Compat.Runtime.Serialization`
shim exists for modern .NET). The helper likely must target .NET Framework 4.x - acceptable since
the agent already runs on Windows, but it adds a .NET runtime to the agent footprint.
- **Security.** `NetDataContractSerializer` instantiates arbitrary embedded .NET types (BinaryFormatter-class
risk). Input here is EPU's own output on the same trusted box, but this must be stated explicitly.
- **Licensing.** Loading Thermo's proprietary assemblies at runtime - confirm this is acceptable for
our deployment (we load DLLs already on the workstation; we do not redistribute them).
- **Version coupling.** The helper must load whatever assembly versions are on that workstation;
behaviour across EPU upgrades needs a check (mitigated by loading the on-disk DLLs, not pinned copies).
- **Interop cost.** Adds a second process/runtime to a Python agent; weigh against the fact that the
current parser already neutralises most plumbing brittleness.

## Alternatives considered (context, not part of this issue)

- **Option A - Smart EPU Open API** (Thermo's supported plugin/automation API): the "proper" upstream,
no filesystem, no XML. Requires Smart EPU enablement/licensing and a plugin host; strategic, separate track.
- **Option B - Athena / DMP data platform**: structured upstream, but the `DMPSession` block is `i:nil`
across all sampled sessions, i.e. DLS runs EPU standalone. Needs infra/licensing, not agent code.
- **Option D - read EPU process memory directly**: rejected - closed-source heap layout, breaks on every
EPU update.

## References

- Format write-up: `smartem-devtools/docs/decision-records/epu-data-structures.md`
- Directory structure: same doc, "EPU Output Directory Structure"
- Current parser: `smartem-decisions/src/smartem_agent/fs_parser.py`; domain models:
`smartem-decisions/src/smartem_common/schemas.py`
- Test dataset: `testdata/datasets/bi37708-28-copy` (EPU 3.9.1.8206, Titan Krios)

Start with the Phase 0 feasibility spike and the referenced testdata/datasets/bi37708-28-copy dataset. Read src/smartem_agent/fs_parser.py, src/smartem_common/schemas.py, and the EPU data-structures decision record, then confirm whether a .NET Framework helper can rehydrate one GridSquare_*.dm with EPU's assemblies. Done means a parity result and a written go/no-go recommendation covering the stated risks.

Written by the indexing model from the issue text.