Census never becomes correct on a continuously-written cluster: every cycle is Superseded, per-bucket cache never seeded — datausageinfo reports 36 objects for a 151k-object bucket (rc.5)

Environment

• RustFS 1.0.0-rc.5 (official image, digest c36b3efe…), Docker, 3 nodes × 2 drives, EC 4+2, TLS
• Same clusters as #6907 / #6958 / #6218. Scanner state was fully reset on 2026-09-02 (bootstrap from scratch, see #6907), so there is no inherited state involved here.
• Workload: one bucket (loki-dati, 150,994 objects / 85.3 GB by mc du) receives continuous Loki writes — new chunks every few seconds, around the clock. RUSTFS_SCANNER_CYCLE=3600 (explicit cadence); everything else default.

Symptom

datausageinfo (and therefore the console and quota accounting) reports the cluster as 6,498 objects / 0.4 GB, with loki-dati at 36 objects / 0.01 GB — off by ~23× on count and ~200× on size — and it has stayed like this since the reset, while the S3 data plane is perfectly healthy:

datausageinfo: objects_total_count 6498 | objects_total_size 0.4 GB | buckets_count 14
               usage_snapshot_complete: true | usage_snapshot_converged: false | scanner_epoch 2 | scanner_cycle 30
  loki-dati        36 objects   0.01 GB      (real: 150,994 objects / 85.3 GB)
  ads-loki       6013 objects   0.02 GB      (real: 2,223 objects — the census counts versions here, fine)

scanner/status on the leader:

cycle_schedule: effective_interval 3600, superseded_cycles 28, superseded_retry_backoff_enabled true
metrics: last_cycle_result "superseded", last_cycle_duration_seconds 44.6, last_cycle_objects_scanned 6215,
         cycles_completed_at [], usage_freshness.last_publication_state "success"

Every cycle since the reset ended superseded; no cycle has ever ended Complete (cycles_completed_at is empty). Each hourly cycle lasts ~45 s and visits ~6k objects — the dirty subset — and the per-bucket usage cache for loki-dati (.rustfs.sys/buckets/loki-dati/.usage-cache.bin) is tiny (xl.meta ≈ 885 B, rewritten every cycle), i.e. it was never seeded with a full walk.

Why (from the rc.5 source)

classify_nsscanner_cycle (crates/scanner/src/scanner_io.rs) returns Complete only when the scan finished and activity_status == Unchanged && dirty_usage_status == Current; any concurrent namespace mutation during the cycle makes it Superseded. Superseded cycles do publish (should_publish_usage_snapshot), but — per the comment in scanner.rs — "a superseded cycle is explicitly observational and cannot replace the authoritative snapshot". On a cluster that is written every few seconds, every cycle is superseded, so:

1. the bootstrap full walk after the reset (≈921k folder/object scans across the nodes) was itself superseded → never became the baseline, per-bucket caches never seeded;
2. every subsequent cycle walks only the dirty subset (~6k objects in 45 s) and publishes it as the census;
3. usage_snapshot_complete: true is reported for a snapshot that covers 4% of the objects.

The activity check appears to be cluster-wide (namespace mutation generation), so a single busy bucket prevents convergence for the whole cluster.

Control experiment — our second cluster, same version and config, where the hot bucket is written in hourly bursts (mirror jobs) with quiet gaps in between: census 22,496 vs real 22,534 objects on that bucket — correct. So the mechanism works as soon as one cycle can finish without concurrent writes; a site that never has such a gap never gets a correct census. This is the census-correctness side of the "never-converged site" problem (#6852): #6896 made the observed snapshot publishable (no more quota fail-close), but the published numbers are the dirty subset, not the usage.

Suggestions

1. A structurally complete walk that was superseded only by concurrent writes is a far better baseline than nothing: let it seed the per-bucket/per-set caches and the authoritative snapshot (marked stale-by-N-mutations if you like) instead of discarding it as observational. Convergence can then proceed incrementally, as it does on the quiet cluster.
2. Scope supersession per bucket/set (a write into bucket A should not invalidate the walk of bucket B), or make the activity proof tolerant of the mutation rate observed during the cycle.
3. Don't report usage_snapshot_complete: true when the snapshot is a dirty-subset publication.
4. Document that clusters with continuous writes cannot currently converge (until the above lands), and that usage-state/reset does not help in that situation.

Full scanner/status / datausageinfo JSON and the per-bucket cache objects available on request; happy to test a patch on this cluster, as usual.