`_purge` only evicts the channel cache on the node that served the request; other nodes keep serving purged doc IDs from `_changes`

Version

Couchbase Sync Gateway 3.3.5.1 EE, running as Capella App Services (multi-node
endpoint behind a load balancer). Server: Couchbase Sync Gateway/3.3.5.1 EE.

Summary

POST /{keyspace}/_purge evicts the purged doc IDs from the channel cache of the
node that handles the request, but not from any other node in the cluster. Those
nodes keep returning the purged IDs from _changes indefinitely. A client that
follows the feed gets a doc ID that GET /{keyspace}/{docid} answers 404 for.

Steps to reproduce

Against a multi-node deployment (2 nodes is enough):

1. Write a few docs into channel X.
2. GET /{keyspace}/_changes?filter=sync_gateway/bychannel&channels=X&active_only=true
   repeatedly until every node has warmed its cache for X.
3. POST /{keyspace}/_purge with {"<docid>":["*"]} for one of them. Response is
   200 and lists the doc as purged.
4. GET /{keyspace}/<docid> — consistently 404 / error code 1404, from every
   node. The document really is gone.
5. Repeat the _changes request from step 2 many times.

Actual

The purged ID is still returned by roughly half the _changes requests — whichever
node did not serve the purge. Ten identical requests in my case:

result-count distribution over 10 identical _changes queries: {21: 5, 7: 5}

Seven is correct. The 21-row responses contain 14 IDs that were purged (two earlier
batches), every one of which _bulk_get reports as missing:

GONE IC5BD4AD2C-…::FO::WSCARRIER::…
GONE RWRiskSummary::IC5BD4AD2C-…::…
… 14 total …
LIVE IC7F353AB0-…::FO::WSCARRIER::…
… 7 total …

The split is stable across hours and survives repeated purges — it is not a
propagation delay.

Expected

Either the purge invalidates the channel cache cluster-wide, or _changes does not
hand out IDs the same deployment answers 404 for.

Analysis

handlePurge does evict the cache (added by #3765, covered by TestPurgeWithChannelCache):

rest/admin_api.go
count := h.collection.RemoveFromChangeCache(h.ctx(), docIDs, startTime)

but that path is in-process only — channelCacheImpl.Remove ranges over
c.channelCaches, this node's map:

db/channel_cache.go
func (c *channelCacheImpl) Remove(...) (count int) {
    c.channelCaches.Range(removeCallback)

Nothing notifies the other nodes, and they cannot infer it from the feed either.
Purge removes the doc with DeleteWithXattrs(key, [_sync, _globalSync]), so the
DCP deletion the other nodes receive has no _sync xattr, and changeCache.DocChanged
returns before doing anything with it:

db/change_cache.go
// If the document has no xattrs, it can not have a _sync xattr
if event.DataType&base.MemcachedDataTypeXattr == 0 {
    return
}

So the eviction is unreachable for every node except the one that served the REST
call. db.Compact uses the same RemoveFromChangeCache call for tombstone purging
and looks to have the same limitation.

Impact

Any client driving reads off _changes has to tolerate 404 on IDs the feed just
gave it. In our case it surfaced as an intermittent, node-dependent test failure that
looked like a product bug for a while.

Workaround

Pass since=<sequence captured before the run> on _changes — stale cache entries
sit below that sequence, so they are filtered out regardless of which node answers.

Start by tracing handlePurge in rest/admin_api.go through RemoveFromChangeCache in db/channel_cache.go, then compare it with changeCache.DocChanged in db/change_cache.go and the db.Compact path. Use TestPurgeWithChannelCache as the existing baseline; done means purged IDs are no longer returned by _changes from any node in a multi-node deployment without breaking cache behavior.