dapr / dapr/components-contrib
JetStream pubsub does not recover after NATS outage; publishes keep failing with "nats: connection closed"
- Dominant language
- Go
- Stars
- 602
- Forks
- 580
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
## Expected Behavior
When the NATS JetStream broker becomes unavailable and then becomes healthy again, the Dapr `pubsub.jetstream` component should automatically recover its connection.
Subsequent publish operations should succeed without requiring a restart of the application pod or the Dapr sidecar.
## Actual Behavior
When NATS becomes unavailable, publishing fails as expected.
However, after NATS becomes healthy again, all subsequent publish attempts continue to fail with:
```text
error when publishing to topic in pubsub pubsub: nats: connection closed
```
The issue persists even after NATS is fully healthy again.
Restarting the application pod / Dapr sidecar immediately restores publishing.
### Environment
- Dapr runtime: `1.17.9`
- Component: `pubsub.jetstream/v1`
- NATS: JetStream
- Deployment: Kubernetes (AKS)
## Steps to Reproduce the Problem
1. Configure Dapr with the following JetStream pub/sub component:
```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: pubsub
spec:
type: pubsub.jetstream
version: v1
metadata:
- name: natsURL
value: "nats://nats.nats.svc.cluster.local:4222"
- name: token
value: ""
- name: streamName
value: "my-stream"
```
2. Start the application and Dapr sidecar.
3. Publish an event and verify that publishing succeeds.
4. Stop all NATS pods.
5. Publish another event and verify that publishing fails while NATS is unavailable.
6. Start NATS again.
7. Wait until all NATS pods are Running/Ready and the Kubernetes service endpoints are healthy.
Example:
```text
nats-0 3/3 Running
nats-1 3/3 Running
nats-2 3/3 Running
```
8. Without restarting the application pod or Dapr sidecar, publish another event.
9. Observe that the publish still fails with:
```text
nats: connection closed
```
10. Restart the application pod / Dapr sidecar.
11. Publish again and observe that publishing succeeds immediately.
## Release Note
RELEASE NOTE: **FIX** JetStream pub/sub connection recovery after NATS broker outage.
Contributor guide
Research direction
Start at the pubsub.jetstream component entry point and reproduce the NATS outage and recovery sequence described in the issue, focusing on the connection lifecycle around publishing. Done means publishes succeed after NATS becomes healthy again without restarting the application pod or Dapr sidecar, and the recovery behavior is covered by an appropriate test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100