funkBuild / funkBuild/timestar
Query dropped series with "CompressedSlice - attempted to read beyond buffer bounds" (response silently INCOMPLETE) — root cause open
- Dominant language
- C++
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 40m
- Merged PRs (30d)
- 6
Description
## Symptom
On 2026-09-03 09:55:33Z production shard 1 dropped two series from one query response:
```
ERROR [shard 1:main] timestar.http - [QUERY] Dropping series '4bec8d70-….deviceData,deviceName=GW64 volumePrevMonth' on shard 1 from result: CompressedSlice - attempted to read beyond buffer bounds [tsm shard_1/tsm/3_2132_d2128.tsm]. The response will be INCOMPLETE.
ERROR [shard 1:main] timestar.http - [QUERY] Dropping series '4bec8d70-….deviceData,deviceName=GW113 batteryVoltage' on shard 1 from result: CompressedSlice - attempted to read beyond buffer bounds [tsm shard_1/tsm/3_2132_d2128.tsm]. The response will be INCOMPLETE.
```
- One occurrence in 30 days of logs (both lines are the same query). Server 1.4.3, up since 2026-09-01 06:35Z, no restart.
- The query was a `latest`-shaped fleet read (193 series / 168 points, 2.3 s). A 4046-series / 2.27M-point scan of the same measurement was running concurrently on the same shard (10.9 s, "Reactor stalled for 39 ms" right after).
- `3_2132_d2128.tsm` is a tier-3 file written by compaction on 2026-09-01 22:32Z (73,863 series). It was not being compacted, rewritten or deleted at the time; the server has no retention policies and no tombstone activity that day.
- **The client was not told.** The HTTP response simply lacked the two series. The only record is the server log line.
### What the error means
`CompressedSlice::boundsCheck` fires inside `ALPDecoder::decode` (every float block goes through ALP). To get there, the block header checks (type byte, `timestampBytes` within block, plausible count) and the timestamp decode all passed — only the value decoder ran out of words. So for that one read the block's recorded `size`/count disagreed with its value section. That is what BOTH a wrong descriptor and wrong bytes look like, and 1.4.3's message distinguished neither.
### Ruled out (2026-09-07 investigation)
- **Persistent corruption.** 1,300+ targeted re-reads of both series (all-time, 10-day chunks over 180 d, per-day over 40 d, 112 six-hour `latest` windows, 744 windows with edges on exact stored timestamps ±1 ns) → zero drops logged, full counts (96/day).
- **File rewritten under a live read.** No tombstone / retention / rewrite / delete on shard 1 that day.
- **Read-elevator view math** (`coalescedDmaRead` / `dispatchPendingDmaReads`): per-request `share(offset − start, size)` is correct; a short read raises its own, distinct error.
- **Index-cache use-after-free.** All five query paths copy the block span out of the LRU before any `co_await` (earlier "use-after-free fix"); the bulk loader is compaction-only and copies too.
- **Concurrency reproduction.** Four rounds of the 09-03 shape (fleet `latest` while a 10-day 4046-series scan runs) → no drops.
- **Timestamp filter off-by-one.** The exact-timestamp boundary sweep above would have hit it.
Root cause **not** established.
## Debug added (1.4.4, #6)
- `readSingleBlockImpl` now re-raises anything thrown after the DMA read with the block's identity and the header facts appended, ahead of the existing file suffix:
```
… [block offset= size= count= time=.. hdr.count= hdr.tsBytes= skip= n= valueBytes=] [tsm ]
```
The aggregator batch path (`aggregateSeriesImpl`) appends the `[block …]` half (no header fields there).
- `readSingleBlockImpl` copies the descriptor before its `co_await` instead of re-reading `indexBlock.size` through the caller's reference afterwards. That was the one way a correct file could produce exactly this failure; every current caller already copied its block list, so this closes the contract rather than a known caller.
- Tests: `ReadSingleBlock_DecodeErrorNamesTheBlock`, `ReadSingleBlock_SurvivesDescriptorMutatedDuringRead`.
## What to look at when it recurs
Search the TimeStar log stream for `[QUERY] Dropping series` (CloudWatch group `/ecs/IoTCore-production`, stream prefix `timestar/`). For each hit, read the `[block …]` suffix:
| Observation | Meaning | Where to look next |
|---|---|---|
| `count` ≠ `hdr.count`, or `size` implausible for `hdr.tsBytes + valueBytes` | The **descriptor** was wrong: stale, mis-parsed, or read from memory that changed under a suspension. | `getFullIndexEntry` / `prefetchFullIndexEntries` parsing; anything still holding a raw `TSMIndexEntry*` into `fullIndexCache` (its `put()` destroys entries on eviction — the copies are what keep it safe). |
| Descriptor and header agree, `skip + n ≤ hdr.count`, decoder still short | The **bytes** were wrong for that read. | Same block re-read cleanly afterwards ⇒ the buffer handed to the decoder was not the file's bytes (elevator `share()` view, buffer lifetime). Fails again on re-read ⇒ the block is bad on disk: pull the file, check the compaction that wrote it (tier-3 ALP re-encode). |
| `skip + n > hdr.count` | Timestamp filter over-counted. | `IntegerEncoder::decode(… startTime, maxTime)` inclusive-bound handling for that window. |
| `hdr.count` etc. absent (only the `[block …]` descriptor half) | Failure came through the aggregator batch path, before or outside the header parse. | `decodeBlockIntoAggregator` / `parseHeaderAndDecodeTimestamps` return `nullopt` on header problems, so a throw here is the value decoder or `decodeBlockAndFold`. |
Also correlate with:
- `[SLOW_QUERY]` lines within ±15 s on the same shard (was a 2.27M-point scan last time) and any `Reactor stalled` line.
- Whether the file was under compaction / had just been written (`Compacted … to tier N`, `Loaded sparse index for … ` timestamps).
- The client side: iot-core's `TIMESTAR_REQUEST_TIMEOUT_MS` is 5 s; a 10-day fleet scan takes ~13 s server-side and stalls the shard, so small queries behind it time out client-side while the server keeps working. Load at the moment of the drop matters.
## Follow-ups this issue tracks
- [ ] Root-cause the next occurrence using the `[block …]` context above.
- [ ] **Surface incompleteness to clients.** A dropped series should reach the caller (a response header / `QUERY_INCOMPLETE` marker), so iot-core can retry or flag instead of rendering a silently short answer as truth — the same failure family as the discovery wrong-empties.
- [ ] **Alert** on `[QUERY] Dropping series` (CloudWatch metric filter → alarm). Today nobody is told.
- [ ] Consider handing out `std::shared_ptr` from `fullIndexCache` so raw-pointer lifetime stops depending on every caller copying first.
Related: PR #5 (day-bitmap membership), PR #6 (this debug), release v1.4.4 (deployed 2026-09-07 02:17Z).
Contributor guide
Research direction
Start with the next `[QUERY] Dropping series` log in the CloudWatch stream and inspect its `[block …]` context, correlating nearby `[SLOW_QUERY]`, `Reactor stalled`, and compaction lines. Trace the relevant path through `readSingleBlockImpl`, `getFullIndexEntry`, `prefetchFullIndexEntries`, and the aggregator entry points according to the descriptor and header observations. Done means the next occurrence has an established root cause and a confirmed remediation path; client incompleteness and alerting remain separate follow-ups.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100