bug(ublk): pooled devices can expose stale placeholder pages after image switch

## Summary

A pooled ublk device can return cached zero pages from its idle placeholder after being rebound to a nonzero business image. In an isolated one-device experiment, buffered reads returned zeros while `O_DIRECT` reads of the same offset returned the correct snapshot content. Flushing only that device's page cache restored correct buffered reads.

We investigated this after intermittent guest kernel panics during concurrent snapshot restores, surfaced to the caller as `timed out waiting for envd`. The stale-page mechanism is reproduced; its connection to the original guest panics is strongly supported but not fully proven. We have not identified the reader that populated the placeholder cache during the original failures or reproduced a complete VM panic in the reduced experiment.

## Affected component

ublk / ublk-daemon, specifically pooled device image switching and host block-device page-cache consistency.

## AgentENV version

- Source inspected: upstream `main` at [`e330917f17d9f5103066df2aea5c2e172d6c6527`](https://github.com/kvcache-ai/AgentENV/commit/e330917f17d9f5103066df2aea5c2e172d6c6527).
- Runtime reproduction used a downstream build. Its `prepare_overlaybd_device`, `idle_released_device`, and `clear_page_cache` functions are byte-for-byte identical to those in the upstream revision above. This is a source comparison, not a claim that the complete upstream revision was runtime-tested.
- Local modifications: present outside these compared functions.

## Host environment

- Host kernel: Linux `6.8.0-101-generic`.
- Architecture: x86_64; host page size: 4096 bytes.
- ublk enabled; device pool enabled.
- Guest: 1 vCPU, 2048 MiB, Linux `6.1.175`; Firecracker v1.15.1-based build.
- Memory restore: OverlayBD memory image exposed through `/dev/ublkbN`, used as Firecracker file-backed memory.
- The tested memory lower was an immutable filesystem-backed OverlayBD commit (`file` lower, empty `repoBlobUrl`), not a registry-backed source.

## Operation and configuration

The original operation was creating sandboxes from committed snapshots. The reduced experiment only starts a separate daemon with one private device; it does not start Firecracker, envd, or readiness probes.

Relevant isolated daemon options:

```text
--enable-pool
--pool-low-watermark 0
--pool-high-watermark 1
--pool-startup-prewarm false
```

Use a separate socket and configuration/cache directory for the experiment. All flushes below target only the experiment's own device.

## Steps to reproduce

Supply a valid read-only OverlayBD image with a known nonzero, 4096-byte-aligned page. We used a 2 GiB memory image and offset `0x7ffce000`.

1. Start the isolated daemon with the pool configuration above.
2. Acquire the image in shared mode (`acquire_overlaybd`, `access_mode: "shared"`, `virtual_size: 2147483648`). Read the selected page using aligned `O_DIRECT` I/O and save its content as `original`; verify it is nonzero.
3. Release the device using `release_overlaybd`. It remains in the pool bound to the empty placeholder.
4. Open a separate read-only FD to this idle device and **keep it open across the subsequent acquire**. This models a concurrent reader whose FD/cache lifetime spans the handover. Verify that an `O_DIRECT` read now returns zeros. In the controlled experiment, wait 200 ms and issue `BLKFLSBUF` on this private device to discard any old business-image read-ahead; then perform a buffered read and verify it returns zeros. The test intentionally fills the placeholder page cache at this point.
5. Acquire the original image again. Verify that the daemon returns the same device ID.
6. Read the selected offset through the buffered path: it still returns zeros. Read the same offset using `O_DIRECT`: it returns `original`.
7. Issue `BLKFLSBUF` on this private device after the switch. The buffered read now returns `original` as well.
8. Close the retained FD, release the device, and shut down the isolated daemon.

The retained reader FD and explicit placeholder read are part of the reproducer, not an observation about which process was responsible in the original workload. The experiment does not assume a particular udev or block-probing implementation.

## Expected behavior

After an acquire completes, buffered and direct reads must refer to the newly bound image. Cache pages populated while the device was idle must not be served as guest memory from a different image.

## Actual behavior

At the same offset of the same private device:

| Binding / read path | First little-endian u64 |
|---|---|
| Original snapshot, `O_DIRECT` | `0x7ffcd067` |
| Idle placeholder, direct and buffered | `0x0` |
| Reacquired snapshot, buffered | **`0x0` (incorrect)** |
| Reacquired snapshot, `O_DIRECT` | **`0x7ffcd067` (correct)** |
| Reacquired snapshot, buffered after device-specific `BLKFLSBUF` | **`0x7ffcd067` (correct)** |

The complete 4096-byte pages matched before the switch, on the direct-read control, and after the flush. The stale buffered page was entirely zero.

An earlier experiment encountered old business-image data while the device was already on the placeholder, so the final controlled experiment explicitly discarded that old cache before filling the placeholder page. Both observations indicate that changing the target alone does not establish a new page-cache identity.

## Logs and diagnostics

Sanitized guest error excerpt from the original restore failures:

```text
BUG: unable to handle page fault
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 7ffce067 P4D 7ffce067 PUD 0
Kernel panic - not syncing: Fatal exception in interrupt
Firecracker exiting successfully. exit_code=0
```

Several failures first faulted in `receive_mergeable`; another first faulted in `kfree` on an ACPI GED call path and subsequently faulted in multiple kernel paths. The common evidence is the missing page-table entry, not a proven virtio-net defect. The tested offset `0x7ffce000` is the PUD-table location indicated by this page-table walk (PUD index 0).

In one representative case, snapshot load returned successfully in approximately 30 ms, the VM resumed, and Firecracker exited after the guest panic approximately 2.3 seconds later. The caller received the envd readiness timeout about 57.7 seconds after that exit. Snapshot load uses demand-backed memory, so its short duration alone does not prove all memory reads were complete.

## Suspected mechanism in current upstream code

- [`idle_released_device`](https://github.com/kvcache-ai/AgentENV/blob/e330917f17d9f5103066df2aea5c2e172d6c6527/storage/ublk-daemon/src/server.rs#L1307) clears the page cache **before** switching to the idle placeholder.
- [`prepare_overlaybd_device`](https://github.com/kvcache-ai/AgentENV/blob/e330917f17d9f5103066df2aea5c2e172d6c6527/storage/ublk-daemon/src/server.rs#L1080) switches the target to the requested image and returns the device, without a post-switch cache invalidation.
- [`clear_page_cache`](https://github.com/kvcache-ai/AgentENV/blob/e330917f17d9f5103066df2aea5c2e172d6c6527/storage/ublk-daemon/src/server.rs#L1603) is currently a best-effort `BLKFLSBUF`.

Clearing before release does not cover cache pages created during the placeholder phase and retained across checkout. Newly created/prewarmed placeholder devices also need a consistent handover; the issue is not necessarily limited to devices previously used by another VM.

A fix needs to consider old in-flight I/O, target switching, and cache invalidation together before the new image is exposed. The flush control in this report demonstrates the cache mismatch; it does **not** establish that adding one unsynchronized flush is a complete fix.

## Reproduction frequency

Intermittent guest failures in the original workload. The reduced one-device experiment reproduced the incorrect buffered read with the explicit retained-reader/cache-fill sequence above. No claim is made that the original VM failure rate or trigger has been reproduced deterministically.

## Regression information

- Last known good: unknown.
- First known bad: unknown.
- The relevant switching/flush functions are still present unchanged in the inspected upstream revision.

## Additional context

The reduced cache experiment does not depend on readiness probes. Separately, [`EnvdInstance::wait_for_ready`](https://github.com/kvcache-ai/AgentENV/blob/e330917f17d9f5103066df2aea5c2e172d6c6527/src/sandbox/envd.rs#L109) only polls health with a deadline; observing Firecracker process exit during this wait would help report the original VM failure promptly instead of masking it as an envd timeout. That would improve diagnostics but would not fix the stale memory contents.

The experiment's private daemon exited successfully and its device was removed. No global page-cache flush was used.

## Pre-submission checklist

- [x] I searched existing open and closed issues and did not find a duplicate.
- [x] I provided a reduced reproduction procedure and stated its limits.
- [x] I removed credentials, private image references, deployment identifiers, hostnames, addresses, and private paths.
- [x] This report concerns a reproduced data-correctness defect; no security vulnerability is being reported.

Start in storage/ublk-daemon/src/server.rs at prepare_overlaybd_device, idle_released_device, and clear_page_cache, tracing target switching, in-flight I/O, and cache handling. Run the isolated pooled-device reproduction with the retained reader FD and buffered/direct reads. Done means a device reacquired for a new image cannot return placeholder data through buffered reads, while the handover remains safe under concurrent access.

Written by the indexing model from the issue text.