asyncapi / asyncapi/bindings

[FEATURE] Kafka Operation Binding: `error-topics` for Retry and DLQ Topic Provisioning

Open
#299 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
No language data
Stars
79
Forks
81
PR merge metrics
No merged PRs in 30d

Description

### Why do we need this improvement?

## Motivation

Enable provisioning complete Kafka infrastructure from AsyncAPI definitions alone, making AsyncAPI the single source of truth, not just for the API contract but for the full cluster topology.

With channel bindings you can already provision main topics. But retry and DLQ topics have to be defined somewhere else. Today there are two options:

- **Define them outside AsyncAPI.** This breaks the single source of truth and causes API drift.
- **Define them as channels and operations.** A single `receive` operation with 5 retry topics and a DLQ would require 6 additional channels and 12 additional operations. And they should not be there: retry and DLQ topics are not part of a service's public API. They are internal infrastructure owned by one consumer.

This proposal closes that gap without touching the contract.

### How will this change help?

This will allow teams define and provision error topics using AsyncAPI as the single source or truth in a non-verbose non-invasive manner.

ZenWave SDK supports generating Terraform from AsyncAPI files for provisioning Kafka OSS, Schema Registry or Confluent Platform and uses this as an extension to provision kafka error topics.

https://www.zenwave360.io/zenwave-sdk/plugins/asyncapi-ops/

### Screenshots

_No response_

### How could it be implemented/designed?

## Proposed Design

### Placement

At the operation binding level, under `bindings.kafka`. Retry and DLQ topics are a consumer concern, the producer has no involvement.

```yaml
operations:
onSomethingHappened:
action: receive
channel:
$ref: '#/channels/some-channel'
bindings:
kafka:
groupId:
type: string
enum: ["some.consumer.service"]
error-topics:
addressTemplate: "${groupId}.__.${channel.address}.${suffix}"
retryTopics: 3
retry:
partitions: 1
replicas: 2
topicConfiguration:
retention.ms: 259200000
dlq:
partitions: 1
replicas: 2
topicConfiguration:
retention.ms: 2592000000
cleanup.policy: ["delete"]
```

### Schema

#### `error-topics`

| Field | Type | Required | Description |
|---|---|---|---|
| `addressTemplate` | `string` | Yes | Template for computing topic addresses. See below. |
| `retryTopics` | `integer` | No | Number of retry topics. Required when `retry` is present. |
| `retrySuffixes` | `array` | No | Explicit suffix names, in order. Defaults to `retry-0`, `retry-1`, … `retry-{retryTopics-1}` when omitted. If provided, its length MUST equal `retryTopics`, and values MUST be unique — tooling should reject the document otherwise. |
| `retry` | `object` | No | Config for retry topics. Omit to skip provisioning. |
| `dlq` | `object` | No | Config for the DLQ topic. Omit to skip provisioning. |

#### `retry` and `dlq`

| Field | Type | Required | Description |
|---|---|---|---|
| `partitions` | `integer` | No | Number of partitions. |
| `replicas` | `integer` | No | Replication factor. |
| `topicConfiguration` | `object` | No | Same schema as channel binding `topicConfiguration`. |

> Environment-specific overrides are out of scope here. The `env-server-overrides` mechanism from #292 applies the same merge semantics and can be extended to cover `retry` and `dlq` if that proposal is adopted.

### Address Template

| Variable | Resolved value |
|---|---|
| `${groupId}` | First value of the `groupId` enum in the operation binding. |
| `${channel.address}` | The `address` of the referenced channel. |
| `${suffix}` | `retry-0`, `retry-1`, … `retry-{N-1}` by default, or the values of `retrySuffixes` in order if provided, plus `dlq`. |

> **`${groupId}` resolution.** `groupId` is a schema object (`type: string, enum: [...]`), not a plain string. Tooling uses the first `enum` value. If absent or no `enum` is defined, resolves to empty string and tooling should emit a warning.

Example — `groupId: some.consumer.service`, channel address `some.domain.someservice.some-event.avro.v0`, `retryTopics: 3`, template `${groupId}.__.${channel.address}.${suffix}`:

```
some.consumer.service.__.some.domain.someservice.some-event.avro.v0.retry-0
some.consumer.service.__.some.domain.someservice.some-event.avro.v0.retry-1
some.consumer.service.__.some.domain.someservice.some-event.avro.v0.retry-2
some.consumer.service.__.some.domain.someservice.some-event.avro.v0.dlq
```

The `.__.` separator is suggested to visually distinguish consumer group from original topic. Tooling can split on `.__.` to recover both parts unambiguously.

#### Naming retry topics explicitly

By default, `retryTopics: N` alone produces conventionally-numbered suffixes (`retry-0` … `retry-{N-1}`). This is enough for a plain fixed-attempt retry loop, but non-blocking retry with backoff (e.g. Spring Kafka's `RetryTopicConfiguration`) conventionally names topics after their delay, not their index. `retrySuffixes` lets you opt into that without changing anything else:

```yaml
error-topics:
addressTemplate: "${groupId}.__.${channel.address}.${suffix}"
retryTopics: 3
retrySuffixes: [retry-5s, retry-30s, retry-5m]
```

resolves to:

```
some.consumer.service.__.some.domain.someservice.some-event.avro.v0.retry-5s
some.consumer.service.__.some.domain.someservice.some-event.avro.v0.retry-30s
some.consumer.service.__.some.domain.someservice.some-event.avro.v0.retry-5m
```

### Reusable Plans via `components`

```yaml
components:
error-topics:
retry:
silver:
partitions: 1
replicas: 2
topicConfiguration:
retention.ms: 259200000 # 3 days
gold:
partitions: 3
replicas: 3
topicConfiguration:
retention.ms: 604800000 # 7 days
dlq:
standard:
partitions: 1
replicas: 2
topicConfiguration:
retention.ms: 2592000000 # 30 days
cleanup.policy: ["delete"]
compliance:
partitions: 1
replicas: 3
topicConfiguration:
retention.ms: 31536000000 # 1 year
cleanup.policy: ["delete"]
```

```yaml
operations:
onSomethingHappened:
bindings:
kafka:
groupId:
type: string
enum: ["some.consumer.service"]
error-topics:
addressTemplate: "${groupId}.__.${channel.address}.${suffix}"
retryTopics: 3
retry:
$ref: '#/components/error-topics/retry/silver'
dlq:
$ref: '#/components/error-topics/dlq/compliance'
```

### 🚧 Breaking changes

No

### 👀 Have you checked for similar open issues?

- [x] I checked and didn't find a similar issue

### 🏢 Have you read the Contributing Guidelines?

- [x] I have read the [Contributing Guidelines](https://github.com/asyncapi/.github/blob/master/CONTRIBUTING.md)

### Are you willing to work on this issue?

Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Begin by locating the existing Kafka channel binding schema, especially its topicConfiguration, and review the env-server-overrides proposal referenced as #292. Compare the proposed operation-level error-topics fields and components references with those existing conventions. Done means the schema supports retry and DLQ provisioning, suffix validation and address resolution as described, without adding public API channels or operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, terraform
Domain
distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.