Opt-in HTTP tap for the raw /dev/diag byte stream
- Dominant language
- Rust
- Stars
- 5.8k
- Forks
- 490
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 23
Description
## Prerequisites
- [x] I have read [CONTRIBUTING.md](https://github.com/EFForg/rayhunter/blob/main/CONTRIBUTING.md).
## What problem does this feature solve or what does it enhance?
The Linux `diag` character device only permits **one consumer at a time in memory-device mode**. As soon as rayhunter holds that handle, any second tool that wants to read `/dev/diag` (QCSuper, other DIAG analyzers, bespoke debugging/capture scripts) is refused by the kernel with `EBUSY`.
Users on a supported device therefore have to choose: run rayhunter's IMSI-catcher analysis **or** run their second tool — never both at the same time. That forces a workflow of "stop rayhunter → run the other tool → restart rayhunter and lose the passive-monitoring gap", which undercuts rayhunter's core value proposition as a persistent background monitor.
This is a known rough edge. It's also a tiny sliver of the much larger multiplexer/unification argument that @trevormjaymes made in [#895 (comment)](https://github.com/EFForg/rayhunter/issues/895#issuecomment-3910593197) — where the proposed end-state is a full `DiagServer` that owns `/dev/diag`, with rayhunter and other clients as fan-out consumers. That is a substantial architectural shift; I'm not proposing to take it on here. I'm proposing the **minimum** change that lets a second DIAG tool coexist with rayhunter today, on one supported device, gated behind an opt-in flag — a stepping stone that could be replaced or absorbed by a fuller multiplexer later without changing the public wire contract much.
## Proposed Solution
Add an **opt-in, off-by-default** HTTP endpoint on the existing axum server:
```
GET /api/diag/stream
```
- Controlled by a new `diag_stream_enabled` boolean in `config.toml`. Default `false`; behavior unchanged for every existing user.
- Also gated on `!debug_mode`, since there is no real `DiagDevice` to tap in debug mode.
- When active, rayhunter's existing single-consumer read of `/dev/diag` is **tee'd** into a bounded `tokio::sync::broadcast` channel *before* parsing. Each HTTP subscriber gets its own `BroadcastStream` view.
- Response is `Content-Type: application/octet-stream`, `Transfer-Encoding: chunked`, `X-DTAP-Version: 1`. Body is the raw bytes rayhunter reads from `/dev/diag` — Qualcomm DIAG memory-device-mode `MessagesContainer` records, byte-for-byte identical to what a direct `/dev/diag` consumer would see. (HDLC framing lives inside each inner message's `data`.)
- Slow subscribers observe `RecvError::Lagged(n)` and drop frames; a single warning is logged per lag. **rayhunter's own parser is never gated on subscriber speed** — that's the key safety property.
- When disabled, the endpoint returns `503 Service Unavailable` with a descriptive message so clients get a clear signal instead of hanging.
The implementation is small (~150 lines of Rust, all behind `diag_stream_enabled`). I made a proof-of-concept and tested it end-to-end on hardware, but per the CONTRIBUTING.md guidance on substantial contributions I'm raising it here before opening a PR. The working branch — if it helps the discussion — is [`lukejenkins/rayhunter:feat/diag-stream-http`](https://github.com/lukejenkins/rayhunter/tree/feat/diag-stream-http).
**Testing already run on an Orbic RC400L:** wire-format smoke test, multi-subscriber byte-for-byte equality check, independent parser validation of captured output (unwrap `MessagesContainer` → HDLC CRC-CCITT verify → DLF decode → 7/7 `LteRrcOtaMsg` parsed cleanly), and a coexistence check confirming rayhunter's own QMDL recording runs uninterrupted while two subscribers are live.
## Alternatives Considered
- **Stop rayhunter, run the other tool, restart** — the status quo. Defeats the point of passive monitoring; creates a gap during which IMSI-catcher activity would go undetected.
- **Ship a separate `diag-multiplexer` daemon that owns `/dev/diag` and offers IPC to rayhunter + other clients** — closer to the full architecture sketched in [#895 (comment)](https://github.com/EFForg/rayhunter/issues/895#issuecomment-3910593197). Cleaner long-term, but a much larger change: new binary, new installer story, new recovery story, breaks the current single-binary model. Worth doing eventually; out of scope here.
- **Export only rayhunter's post-decode PCAP stream** — lossy. Rayhunter's PCAP conversion only covers a subset of log codes; a second tool running DIAG-level analysis needs the raw bytes. Also doesn't help tools that depend on DIAG request/response commands rather than passive log reads.
- **Background-export each read to a file, let a second tool tail the file** — works, but: disk I/O on small-flash devices, file-rotation complexity, and a tailer sees writes at rayhunter's read cadence with whatever buffering the filesystem adds. An HTTP stream is cheaper to reason about and doesn't touch storage.
- **Add a compile-time feature flag instead of a runtime config flag** — runtime is preferable: same binary works for everyone; users don't need to rebuild to enable it, and distributions don't have to pick a single default.
## AI-assisted contribution disclosure
~~This issue body was written by me.~~ The patch itself was drafted with AI assistance (Claude); design, tap insertion point, backpressure semantics, 503 gating, `X-DTAP-Version`, config-flag naming, and wording were reviewed line-by-line. Hardware validation on RC400L was run and reviewed by me.
Contributor guide
Research direction
Review CONTRIBUTING.md, config.toml, and the existing axum server first; trace the current /dev/diag read path and where raw bytes are available before parsing. Done means an opt-in endpoint streams raw bytes with the stated headers and 503 behavior when disabled, without slowing rayhunter’s parser; validate with the wire-format, multi-subscriber, parser, and coexistence checks described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100