drain_on_version_upgrade evicts older-version envoy websockets before stopping their actors (ee-20260903-153931, rivetkit 2.3.2)

Summary

With drain_on_version_upgrade: true on a serverless pool, pegboard_envoy_drain_older_versions closes the older-version envoy websockets in the same millisecond it logs "draining older envoy versions", with outgoing_close_reason=ws.eviction#<ray> and err=eviction: The websocket has been evicted and should not attempt to reconnect. No CommandStopActor reaches the actors resident on those envoys first and the pool's drain_grace_period (35s) is not honoured. Every resident actor loses its sqlite VFS mid-request: rivetkit-napi logs hundreds of failed to persist actor save tick: envoy channel closed, every statement fails with failed to step sqlite execute statement: disk I/O error, and the actors are only restarted (next generation) ~25s later.

docs-internal/engine/actor-rescheduling.md (synced to OSS in b92d85a8, 2026-09-04) describes the intended path as ToEnvoyConnClose -> envoy replies ToRivetStopping -> evict_actors migrates actors gradually. That is not what this bundle does; the actors get no stop at all.

Environment

• Engine: Rivet Enterprise bundle ee-20260903-153931 (manifest gitSha b1b26550036b5a5c8e67720be71e25dfd537271a, generated 2026-09-03T12:45Z), self-hosted on EKS with FoundationDB (cluster healthy, fully replicated, 0 engine/FDB restarts).
• Runner: rivetkit 2.3.2, native (@rivetkit/rivetkit-napi), serverless pool default, drainGracePeriod: 35, drainOnVersionUpgrade: true, requestLifespan: 86400, actor_stop_threshold: 600000, envoy protocol v5.
• Runners are Kubernetes pods; a rollout starts new pods with a higher RIVET_ENVOY_VERSION while old pods are still serving.

Timeline (UTC, 2026-09-16, one occurrence; the pattern repeated 13 times in 6h)

Engine (rivet-engine):

04:03:05  CommandStartActor for actor dpnobs2b… gen 47 sent to envoy 559b3882 (older version 1789527514)
04:03:49  CommandStartActor for actor xwavxced… gen 17 sent to envoy 559b3882
04:04:11.109 pegboard_runner_pool_metadata_poller / pegboard_envoy_drain_older_versions:
            "removed obsolete older-version serverless Envoy allocation-index entries" new_version=1789529933 removed_index_entry_count=4
            "draining older envoy versions due to drain_on_version_upgrade" new_version=1789529933 older_envoy_count=2
04:04:11.109 "envoy websocket closed" envoy_key=559b3882 lifecycle_kind=err incoming_close_code=None
            outgoing_close_code=Some(1000) outgoing_close_reason="ws.eviction#xkl7agiu…"
            err="eviction: The websocket has been evicted and should not attempt to reconnect" lifetime_seconds=1745
04:04:11.113 same for envoy_key=a4067af8 (lifetime 1654s)
04:04:21  "requested expiration for stale older-version Envoy" (last_ping 04:04:10.8)
04:04:33-38 "slow websocket open wait", "no responders for gateway message after retry budget exhausted, aborting" x3 (gateway still routing to the evicted envoys)
04:05:00  "created a new in flight entry after hibernation, reopening websocket"

Runner on envoy 559b3882 (rivetkit-napi):

04:04:10.6  actor dpnobs2b… gen 47 handling an action normally
04:04:11.109 "websocket session ended" reason=close session_duration_ms=1745279
04:04:11.109 "envoy stopped"
04:04:11-33  977 x "failed to persist actor save tick" error="persist actor state deltas to kv / Caused by: envoy channel closed"
             "failed to persist actor state" description=schedule_cancel, "failed to persist last pushed actor alarm"
             "sqlite flush commit failed", "sqlite sync failed", "sqlite atomic commit failed", "sqlite xRead failed to resolve pages"
             every user statement: RivetError group=core code=internal_error
               "actor dpnobs2b… generation 47 key … / Caused by: failed to step sqlite execute statement: disk I/O error"
               (also "cannot rollback - no transaction is active")
04:04:35   SIGTERM from Kubernetes only arrives here; "shutdown: draining"
04:04:36   next generation of the same actors starts on new-version envoys ("sqlite initial page preload request" gen 48/134)

No CommandStopActor and no actor lifecycle transition Started->SleepGrace for those actors between the drain log and the close. Compare the Kubernetes SIGTERM path on the same bundle, which does work: the envoy reports stopping, the engine sends CommandStopActor{reason: SleepIntent} to every resident actor, they go Started->SleepGrace->SleepFinalize->Terminated, and the socket closes with incoming_close_reason=envoy.shutdown, outgoing=ws.closed.

Expected

drain_older_versions should stop the resident actors with the configured drain_grace_period (or the evict_actors pacing) before the websocket is closed, so state persists and the actors are rescheduled with their last state, the same as the SIGTERM drain.

Impact

Every deploy of a new runner version loses the in-flight work of every actor resident on the previous version. We have turned drainOnVersionUpgrade off as a workaround, which brings back the pre-drain behaviour where Kubernetes-owned pod kills could wedge actors in actor_ready_timeout limbo.

Question

Is this fixed in a bundle after ee-20260903-153931? The 2026-09-04 OSS sync (b92d85a8) added evict_actors.rs and the rescheduling doc; if the EE drain now goes through ToEnvoyConnClose -> ToRivetStopping -> evict_actors, a pointer to the first bundle carrying it would let us re-enable the drain.

Start with docs-internal/engine/actor-rescheduling.md, the OSS-synced evict_actors.rs path, and pegboard_envoy_drain_older_versions. Trace whether the drain follows ToEnvoyConnClose through ToRivetStopping before closing the websocket, comparing it with the Kubernetes SIGTERM path. Done means resident actors receive an orderly stop, the drain grace period is honored, state persists, and the behavior is verified in a later bundle.