alunduil / alunduil/blog.alunduil.com

The backoff hint that wasn't there: chasing google.rpc.RetryInfo through Firestore

Open
#329 1 comment 0 reactions 0 assignees View on GitHub
idea
Dominant language
Astro
Stars
1
Forks
1
Avg merge
6h 34m
Merged PRs (30d)
118

Description

## Spark

Adding `Retry-After` to 429 and 503 responses in genshin.dungeon.studio was
meant to end with forwarding the backend's own backoff hint. RFC 9110 wants a
real number in that header. Google's RPC vocabulary defines
`google.rpc.RetryInfo` for exactly this. Firestore surfaces gRPC errors. The
hint should be sitting right there.

An early revision of dungeon-studio/genshin.dungeon.studio#952 forwarded it,
then reverted to fixed per-status defaults — 429 to 30s, 503 to 5s — because
reading the hint needed a hand-rolled shim over `GoogleError.statusDetails`,
typed as an opaque `protobuf.Message[]`. The question was reopened as #955 with
two gates: evidence Firestore populates `RetryInfo` in practice, and a typed
proto source to decode it through. Both came back negative, and a third finding
made the question unreachable anyway.

## Why it could be interesting

The standard assumes a channel that this stack never fills, and the fixed
default is not a compromise — it is the correct answer once the hint provably
isn't there. Every finding is citable:

- Firestore's [Understand error codes][error-codes] never mentions `RetryInfo`.
For `RESOURCE_EXHAUSTED` it says only "retry with exponential backoff."
- `@google-cloud/firestore@8.7.1` contains zero references to `RetryInfo` or
`retryDelay` in `build/`. The first-party client for this exact backend backs
off on the fixed schedule in `firestore_client_config.json` and never looks
for the hint. If the backend sent it routinely, that client would be its
first consumer.
- Against the Firestore emulator, a server-side gRPC error carries an empty
`grpc-status-details-bin`, so `statusDetails` is `undefined`.
- `google-gax@6.0.2` ships `build/protos/status.json`, which does define
`google.rpc.RetryInfo`, but no generated `status.d.ts` beside it — unlike
`operations.json` and `iam_service.json`, which both have one.
- The error never reaches a handler that could read the hint. `apps/api`'s
`err instanceof GoogleError` gate is always false at runtime: `google-gax`
mutates the `@grpc/grpc-js` `ServiceError` in place instead of boxing it, and
two copies of `google-gax` resolve in the workspace. Filed as #1427; until it
lands, every Firestore transient failure returns a bare 500 with no
`Retry-After` at all.

The last one is the sting — the investigation into a missing nicety turned up a
defect in the feature it was meant to improve.

## Open questions

- [ ] Does #1427 land before the post, changing the ending from "and the
handler doesn't even run" to "and fixing that was the real win"?
- [ ] How much RFC 9457 context the post carries. `application/problem+json`
landed separately (#461, PR #1173); extension members were closed not
planned (#520) with no members in use. Neither is load-bearing for the
hint hunt, so probably a sentence, not a section.

## Source material

- dungeon-studio/genshin.dungeon.studio: #512 and PR #952 (fixed-default
`Retry-After`, shipped) · #955 (the investigation and its three findings,
closed not planned) · #1427 (`instanceof GoogleError` never true, open)
- `apps/api/src/http/retry-after.ts` — the shipped fixed-default table
- [RFC 9110 §10.2.3][rfc9110] (`Retry-After`) · [RFC 9457][rfc9457] (Problem
Details)
- — DynamoDB
US-EAST-1 outage postmortem, for why backoff hints matter

[error-codes]: https://docs.cloud.google.com/firestore/native/docs/understand-error-codes
[rfc9110]: https://www.rfc-editor.org/rfc/rfc9110#section-10.2.3
[rfc9457]: https://www.rfc-editor.org/rfc/rfc9457

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.