firebase / firebase/firebase-admin-node

[FR] Expose retry-attempt observability (hook/event) — retried 503s and per-attempt latency are invisible to callers

オープン
#3,214 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
api: messaging
主要言語
TypeScript
スター
1.7k
フォーク
419
平均マージ
3日 10時間
マージ済み PR(30日)
16

説明

### Is your feature request related to a problem?

The HTTP clients retry transient failures — per the code comment in `api-request`, "Retries up to 4 times on connection reset and timeout errors as well as 503 errors" — which is great, but the retry lifecycle is completely invisible to the caller. You observe only the final result and total elapsed time.

In production this matters a lot for FCM:

- A `sendEach()` call that "took 15s" is indistinguishable from one that made three 5s attempts. Capacity planning, SLO attribution, and incident analysis need to tell these apart.
- 503s that the SDK retries away never appear anywhere. During an FCM backend hiccup, our error rate looked flat while the wire was full of retried 503s — we only learned this after instrumenting below the SDK.

### What we did as a workaround (and what it revealed)

We attached listeners at the HTTP/2 layer to record, per attempt, the `:status` header the SDK already receives, plus per-attempt request→response duration. Two things became visible immediately:

1. Swallowed 503s during backend episodes (invisible at the SDK surface).
2. Batches with ~15s total latency decomposed cleanly into 3 × ~5s attempts — i.e. the tail was retry behavior, not slow single requests.

We validated the accounting at scale: in a ~650k-message load run at ~2,000 rps, wire-level attempt counts reconciled exactly with SDK-level result counts (596,663 = 596,663).

Monkey-patching works but is version-fragile and clearly not the intended way.

### Describe the solution you'd like

Any of these would solve it (in rough order of preference):

1. An `onRetryAttempt(info)` callback / EventEmitter on the messaging or app options, with `{ attempt, statusCode?, errorCode?, elapsedMs, willRetry }`.
2. Attempt metadata attached to the final response/error (e.g. `attempts: [{status, elapsedMs}, ...]`).
3. At minimum, a debug logging hook for retry decisions.

### Describe alternatives you've considered

- Runtime-patching the HTTP/2 request path to observe response headers (works, but couples us to SDK internals).
- `enableLegacyHttpTransport()` + external proxy metrics (gives up HTTP/2).

### Additional context

Related: #1615 (custom RetryConfig) — configuration and observability of the same mechanism. Verified against 12.7.0 and 14.1, on Node 16 and 24.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start with the retry behavior referenced in the `api-request` code comment and review the messaging or app options discussed in the request, along with related issue #1615. Done means callers can observe retry attempts, status or error details, elapsed time, and whether another retry will occur, without patching HTTP/2 internals.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
nodejs, typescript
領域
api, observability
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
静か
明瞭さ
説明が足りない
初心者へのやさしさ
28/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。