bug: write-through eviction takes the one path with no offload protection

What happened:

With leader.offload.enabled: true and leader.offload.onEvict unset -- write-through, which is what
an absent onEvict renders -- memory eviction does not wait for the offload to complete. An
object still sitting in an unflushed bucket has its only replica evicted, and the object is gone.

Four runs of a bare-process harness, 32 objects of 4 MiB each, read back by digest:

run	bucket pair	not readable	memory eviction over the whole run
B2	256Mi / 500 (upstream default)	7	never attempted
B3	256Mi / 500 (upstream default)	11	3 of 3 succeeded, 11 keys freed
B5	10Mi / 10	2	3 of 3 succeeded, 11 keys freed
B4, with onEvict: true	256Mi / 500	0	2 of 2 succeeded, 0 keys freed

No run produced a single object that read back with the wrong contents, so every figure above is an
object that could not be read at all.

Only B3 and B5 are evidence for this issue. B2 attempted no eviction at any point in its run, so
whatever made those 7 objects unreadable, an eviction was not it. That cause is unknown and has not
been investigated. It is recorded rather than dropped because an unexplained loss on a run with no
eviction may be a second defect, not a weaker instance of this one.

B5 also rules out tuning as the remedy. Its bucket pair is smaller than the 16Mi / 64 this
operator renders today, so its flush was more frequent and its unflushed window narrower than a
backend has now, and it still lost objects.

B4 is the deferred path, and its row is the mechanism in miniature: eviction succeeded twice and
freed nothing, which is what a reference count pinning the memory replica looks like from the
outside.

What you expected to happen:

An object written to a backend with a disk tier is either on disk or in memory. Evicting the memory
replica of an object whose disk write has not landed destroys the only copy that exists.

How to reproduce it (as minimally and precisely as possible):

Create a KVCacheBackend with a local disk tier and leader.offload.enabled: true, leaving
onEvict unset. Write objects until memory eviction starts, and read them all back. The objects
whose bucket had not closed at eviction time do not come back.

The cause, and why no operator setting reaches it:

Read from the Mooncake source at the pinned v0.3.13.post1 and recorded in full in a comment on this
issue, with line-level citations. In short:

• the store does implement "do not evict until the offload has landed" -- a disk-replica check, a
  reference count that pins the memory replica from the moment it is queued until the client reports
  back, and a skip-this-cycle fallback whose own comment says it exists "to prevent silent data loss
  when the queue is unavailable";
• all three sit below if (!offload_on_evict_) return evict_replicas(...), so write-through
  returns before reaching any of them;
• write-through's own offload push happens once, on the put path, and a push that fails leaves no
  reference count and no retry behind.

The protection is therefore not a setting but the lifetime of a reference count, and which branch
maintains it is selected by offload_on_evict. No environment variable or flag this operator
renders can change that. The question "is there an upstream switch that makes eviction wait" is
answered: there is none.

One observation at the thresholds this operator renders today:

B5's harness was re-run with the bucket pair as its only change, set to the 16Mi / 64 that
pkg/worker/kvcache/mooncake/member_workload.go renders. It lost no objects -- and the reason is
not that the race went away. Memory eviction attempted 352 times and succeeded zero times, freeing
nothing at all, so nothing was ever evicted for an unflushed offload to race against. The same run
began refusing writes earlier than B5 did: at the 33rd object of the second wave against B5's
37th.

A larger bucket is harder to fill, so the offload does not complete, so the reference on the memory
replica is never released, so memory cannot be reclaimed. At today's thresholds the visible failure
on this path may therefore be a store that stops reclaiming memory rather than one that loses
objects. This is one observation on a bare-process harness, not a conclusion, and it says
nothing about how a real deployment behaves. It is recorded because a read of 0 objects lost on
this path is not evidence that the defect is gone.

What has been done here, and why it does not close this:

Admission now refuses leader.offload.enabled: true with onEvict unset, so the combination that
has no protection at all cannot be created through this operator. That is a breaking API change and
is deliberate: it removes the mode rather than documenting it.

That makes the defect unreachable, not repaired. Write-through is still unprotected in the
store, and an administrator reaching it by any other route -- a hand-written client, a different
operator, a future field that re-opens the path -- still loses objects. This issue stays open to
carry that half.

What would close this:

The store stops evicting the sole replica of an object whose offload has not landed. That is an
upstream change; the comment on this issue names the three shapes it could take. Nothing in this
repository can produce it.

What does NOT close this:

• Documenting the mode. An administrator who reads the docs and chooses write-through still
  loses objects.
• The onEvict: true reading in #200. Different mode. Source now says that mode holds the three
  protections, but that is a reading of the code and not a measurement of the same race.
• A unit test over the rendered flags. The flags are correct; what they select is what loses
  data.
• Lowering the bucket thresholds further. #331 already narrowed the window. Tuning changes how
  often this reproduces, not whether it can.
• Recommending onEvict: true as a default. It leaves anyone who deliberately chooses
  write-through in the same place, which refusing the combination does not.

Environment:

• Kubernetes version (use kubectl version): measured on both a three-node cluster and a single
  Docker host
• GPUStack version: n/a
• GPUStack Operator version: the operator at the time of the #200 investigation
• Cloud provider or hardware configuration: not fabric-dependent
• OS (e.g: cat /etc/os-release): n/a
• Kernel (e.g. uname -a): n/a
• Install tools: helm chart
• Accelerator preflight: not involved
• Others: Mooncake pinned at v0.3.13.post1

/kind bug
/area worker

Start with pkg/worker/kvcache/mooncake/member_workload.go and trace the admission path and rendered offload thresholds, then inspect the pinned Mooncake v0.3.13.post1 source and the issue's cited comment. Done requires the store to protect the sole replica during write-through offload; the issue states that no repository-local change can produce this upstream fix.