dapr / dapr/components-contrib

JetStream pubsub does not recover after NATS outage; publishes keep failing with "nats: connection closed"

Open
#4,563 1 comment 1 reaction 0 assignees View on GitHub
kind/bug
Dominant language
Go
Stars
602
Forks
580
Avg merge
4d 9h
Merged PRs (30d)
6

Description

## Expected Behavior

When the NATS JetStream broker becomes unavailable and then becomes healthy again, the Dapr `pubsub.jetstream` component should automatically recover its connection.

Subsequent publish operations should succeed without requiring a restart of the application pod or the Dapr sidecar.

## Actual Behavior

When NATS becomes unavailable, publishing fails as expected.

However, after NATS becomes healthy again, all subsequent publish attempts continue to fail with:

```text
error when publishing to topic in pubsub pubsub: nats: connection closed
```

The issue persists even after NATS is fully healthy again.

Restarting the application pod / Dapr sidecar immediately restores publishing.

### Environment

- Dapr runtime: `1.17.9`
- Component: `pubsub.jetstream/v1`
- NATS: JetStream
- Deployment: Kubernetes (AKS)

## Steps to Reproduce the Problem

1. Configure Dapr with the following JetStream pub/sub component:

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: pubsub
spec:
type: pubsub.jetstream
version: v1
metadata:
- name: natsURL
value: "nats://nats.nats.svc.cluster.local:4222"
- name: token
value: ""
- name: streamName
value: "my-stream"
```

2. Start the application and Dapr sidecar.

3. Publish an event and verify that publishing succeeds.

4. Stop all NATS pods.

5. Publish another event and verify that publishing fails while NATS is unavailable.

6. Start NATS again.

7. Wait until all NATS pods are Running/Ready and the Kubernetes service endpoints are healthy.

Example:

```text
nats-0 3/3 Running
nats-1 3/3 Running
nats-2 3/3 Running
```

8. Without restarting the application pod or Dapr sidecar, publish another event.

9. Observe that the publish still fails with:

```text
nats: connection closed
```

10. Restart the application pod / Dapr sidecar.

11. Publish again and observe that publishing succeeds immediately.

## Release Note

RELEASE NOTE: **FIX** JetStream pub/sub connection recovery after NATS broker outage.

Contributor guide

Open the contributing guide

Research direction

Start at the pubsub.jetstream component entry point and reproduce the NATS outage and recovery sequence described in the issue, focusing on the connection lifecycle around publishing. Done means publishes succeed after NATS becomes healthy again without restarting the application pod or Dapr sidecar, and the recovery behavior is covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.