GoogleCloudPlatform / GoogleCloudPlatform/gcsfuse
Feature request: client-side write coalescing / configurable finalize-debounce for hot (repeatedly-rewritten) objects
- Dominant language
- Go
- Stars
- 2.3k
- Forks
- 510
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 35
Description
> **Note:** This is a **feature request / enhancement**, not a bug. The repository only provides a bug-report issue template, so this follows that template's structure where it applies (System & Version below) and adds the standard feature-request sections.
## Is your feature request related to a problem? Please describe.
Applications that repeatedly rewrite a *single* object — progress/status files, lock files, counters, config-state files — hit the GCS per-object mutation limit (~1 mutation/sec to a single object name) and get:
```
rpc error: code = ResourceExhausted desc = The object / exceeded the rate limit for object mutation operations (create, update, and delete). Please reduce your request rate. See https://cloud.google.com/storage/docs/gcs429.
```
This is the same error reported in #2783 (labeled `known-issues` / `grpc-known-issues`, closed as "retries were added"). Retry-with-backoff smooths *bursts*, but under sustained churn the object simply cannot accept more than ~1 mutation/sec, so:
- callers that keep retrying block for the full backoff window (and, with `ignore-interrupts` on, can wedge the FUSE op in uninterruptible state); and
- callers that bound `max-retry-attempts` (precisely to avoid that wedge) surface the 429 to the application as an I/O error.
Streaming writes only helps when the file is held open and appended sequentially — one finalize on `close()`. The real-world offenders don't do that: they `open(O_TRUNC)` / write / `close()` per tick, or read-back-then-write, each of which finalizes the object (one mutation per tick). No current flag changes that behavior.
Context: we host a large fleet of WordPress/Drupal sites on GCSFuse. Third-party plugins we don't control do this routinely — for example a theme demo-importer that rewrites `uploads/_log/import_progress.txt` after every imported item, or Wordfence rewriting `wflogs/config-synced.php`. We can advise customers, but we can't patch every plugin.
## Describe the solution you'd like
An opt-in, client-side write-coalescing / finalize-debounce for hot objects:
- A configurable minimum interval between finalizations of the same object (e.g. `write.min-finalize-interval` / `write.coalesce-window`), default off.
- When set, GCSFuse debounces `close()`/`fsync()`-driven finalizations on a rapidly-rewritten object so that N rewrites within the window collapse to ≤1 GCS mutation, with a trailing flush after the window drains.
- A clear, documented durability trade-off: within the window the latest bytes are not yet on GCS and not yet visible to other readers — the same trade-off streaming writes already documents for `fsync`, made explicit and bounded by the window.
- Optionally scoped by a path glob (e.g. only coalesce `**/*.tmp`, `**/wflogs/**`, `**/*_progress.txt`) so it never touches normal writes.
## Describe alternatives you've considered
- **App-side changes** (write transient files to local disk, append-and-hold-open, throttle): correct where we control the code; impossible for third-party plugins.
- **Widening retries**: re-introduces the wedged/uninterruptible-op behavior; trades an error for a hang.
- **`write.finalize-file-for-rapid`**: Rapid-storage-class only; not applicable to standard buckets.
## System & Version
- **OS:** All
- **Platform:** GKE / GCE VM (applies to any platform)
- **Version:** confirmed on gcsfuse v3.9.2 (affects v2.0.1 onwards), `--client-protocol=grpc`, streaming writes default-on
## Steps to reproduce
1. Mount any standard (non-Rapid) bucket with `--client-protocol=grpc` (streaming writes default-on).
2. Repeatedly rewrite a single file faster than once per second, e.g. an `open(O_TRUNC)` / write / `close()` loop on one path, or `fio` re-writing over the same object (as in #2783).
3. Observe `ResourceExhausted … exceeded the rate limit for object mutation operations` in the logs; with bounded `max-retry-attempts` this surfaces to the application as an I/O error.
## Additional context
- Supersedes the underlying need behind #2783, which reported this error but was closed on retry alone. There is currently no open issue tracking a reduction of the *mutation rate itself*.
Contributor guide
Research direction
Start by reproducing the ResourceExhausted error with the described gcsfuse mount and repeated open/write/close loop. Trace the close() and fsync() finalization paths, along with the existing streaming-write and retry behavior. Done means an opt-in, bounded coalescing behavior with documented durability semantics, coverage for repeated rewrites, and configuration documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud, grpc
- Domain
- cloud, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100