[Experimental] Allow delivery retry backoff intervals to be capped
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 631
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 6
Description
**Description**
Resource owners can configure the number of delivery retries, the backoff policy, and the initial delay, but they cannot set a maximum interval between normal delivery attempts.
This makes a large retry count impractical with exponential backoff. For example, with `backoffDelay: PT1S`, the calculated interval is about 12 days after 20 retries and about 34 years after 30 retries. If a downstream service recovers during one of these waits, the event is not retried within a useful time. Resource owners therefore have to choose between stopping retries too soon during a long outage and allowing retry intervals to grow beyond an operationally useful range. Very large retry counts can also overflow the duration used by the delivery implementation.
Retry implementations commonly let users cap an increasing backoff interval. For example, Kubernetes exposes [`Backoff.Cap`](https://pkg.go.dev/k8s.io/apimachinery/pkg/util/wait#Backoff), and `cenkalti/backoff` exposes [`ExponentialBackOff.MaxInterval`](https://github.com/cenkalti/backoff/blob/v5.0.3/exponential.go). Knative Eventing should offer the same control for delivery retries.
**Design Overview**
Add an optional `backoffMax` field to `DeliverySpec`. While the feature is experimental, cluster operators enable the field through the `delivery-backoff-max` flag in the `knative-eventing/config-features` ConfigMap. The field accepts a positive ISO 8601 duration and caps intervals calculated from `backoffDelay` and `backoffPolicy` for both linear and exponential backoff.
`backoffMax` has the same meaning for Subscription and Trigger delivery. A Subscription's delivery overrides its Channel's delivery defaults, while a Trigger's delivery overrides its Broker's delivery defaults.
This Subscription starts with a one-second exponential backoff and prevents later delivery attempts from waiting longer than ten minutes:
```yaml
apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
name: orders-to-processor
namespace: default
spec:
channel:
apiVersion: messaging.knative.dev/v1
kind: InMemoryChannel
name: orders
subscriber:
ref:
apiVersion: v1
kind: Service
name: order-processor
delivery:
retry: 100
backoffPolicy: exponential
backoffDelay: PT1S
backoffMax: PT10M
```
The same delivery configuration applies when the service receives events through a Trigger:
```yaml
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
name: orders-to-processor
namespace: default
spec:
broker: default
subscriber:
ref:
apiVersion: v1
kind: Service
name: order-processor
delivery:
retry: 100
backoffPolicy: exponential
backoffDelay: PT1S
backoffMax: PT10M
```
`backoffMax` applies only to intervals calculated from `backoffDelay` and `backoffPolicy`. A delay requested by a subscriber through a `Retry-After` response header remains controlled by the existing `retryAfterMax` field. When `backoffMax` is omitted, existing retry behavior remains unchanged for representable durations. Calculations that exceed `time.Duration` should saturate instead of overflowing to an invalid duration. Channel delivery defaults must preserve `backoffMax` when applied to a Subscription, and Broker delivery defaults must preserve it when used for a Trigger.
**Exit Criteria**
- With the feature flag enabled, resource owners can set a valid, positive `DeliverySpec.backoffMax` value for Subscription and Trigger delivery.
- Linear and exponential intervals calculated from `backoffDelay` and `backoffPolicy` never exceed the configured maximum.
- Invalid, zero, and negative values are rejected, and setting the field while the feature is disabled is rejected.
- Channel delivery defaults preserve `backoffMax` when applied to a Subscription, and Broker delivery defaults preserve it when used for a Trigger.
- Eventing implementations that accept `backoffMax` apply the same cap instead of silently ignoring the field.
- `Retry-After` remains independently controlled by `retryAfterMax`.
- Omitting `backoffMax` preserves existing retry behavior, and large durations or retry counts do not overflow.
**Experimental Feature Flag Name**
`delivery-backoff-max`
**Experimental Feature Stages**
The following is the proposed plan for moving through the stages of the [experimental features process](https://github.com/knative/eventing/blob/main/docs/experimental-features.md):
- Alpha: _(Target v1.24)_
- [ ] Add `DeliverySpec.backoffMax`, the feature flag and validation, capped retry calculations, and support for Channel-to-Subscription and Broker-to-Trigger delivery. Upstream implementation PR: [#9279](https://github.com/knative/eventing/pull/9279).
- [ ] Add unit tests for linear and exponential caps, invalid values, disabled feature behavior, large durations and retry counts, independence from `Retry-After`, and Broker-to-Trigger delivery propagation. [#9279](https://github.com/knative/eventing/pull/9279)
- [ ] Add E2E tests for the documented Subscription workflow and the Trigger workflow using the default Broker implementation. [#9279](https://github.com/knative/eventing/pull/9279)
- [ ] Update the Eventing API reference and delivery documentation. [#9279](https://github.com/knative/eventing/pull/9279)
- [ ] Add user documentation to `knative/docs`. Docs PR: [knative/docs#6686](https://github.com/knative/docs/pull/6686).
- Beta: _(Minimum of one release after Alpha)_
- [ ] Incorporate feedback from Alpha users and implementations.
- [ ] Before enabling the feature by default, add support to the maintained implementations that expose `DeliverySpec`, including Kafka Broker, KafkaChannel, NATS Broker, and NatsJetStreamChannel. NATS implementation PR: [knative-extensions/eventing-natss#802](https://github.com/knative-extensions/eventing-natss/pull/802). Track Kafka support in a follow-up issue or PR.
- [ ] Enable the feature by default.
- [ ] Maintain more than 80% unit test coverage for the feature.
- [ ] Expand E2E coverage across Subscription and Trigger delivery and the main Eventing implementations.
- [ ] Stabilize and improve user documentation.
- Stable: _(Minimum of two releases after Beta)_
- [ ] Remove the `delivery-backoff-max` feature flag and experimental validation.
- [ ] Document `DeliverySpec.backoffMax` in `knative/specs`, including whether support is normative or non-normative.
- [ ] Add conformance tests if required by the stable specification.
**Affected WG**
- Eventing WG
- Eventing Kafka WG
**Additional Context**
- Resources
- [Experimental Features Process](https://github.com/knative/eventing/blob/main/docs/experimental-features.md)
- [DeliverySpec RetryAfter feature track](https://github.com/knative/eventing/issues/5811)
- [DeliverySpec RetryAfter implementation](https://github.com/knative/eventing/pull/5813)
- Prior art
- [Kubernetes `Backoff.Cap`](https://pkg.go.dev/k8s.io/apimachinery/pkg/util/wait#Backoff)
- [`cenkalti/backoff` `ExponentialBackOff.MaxInterval`](https://github.com/cenkalti/backoff/blob/v5.0.3/exponential.go)
Contributor guide
Research direction
Start with DeliverySpec and the delivery-backoff-max feature in the config-features ConfigMap, then trace Subscription/Channel and Trigger/Broker propagation and the retry calculation paths described by the issue. Compare the requested unit, E2E, and documentation coverage with implementation PR #9279 and docs PR #6686; done means every listed exit criterion is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- api, backend-api-design, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100