influxdata / influxdata/telegraf
feat(inputs.nats_consumer): add durable consumer support for JetStream
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Use Case
When running Telegraf with the NATS Consumer input plugin against JetStream streams, restarting Telegraf causes the ephemeral push consumer to be destroyed. On restart, a new ephemeral consumer is created and — depending on the deliver policy — messages are either replayed from the beginning of the stream or missed entirely. This makes Telegraf unsuitable for production JetStream workloads where at-least-once delivery and restart resilience are required.
Durable consumers persist on the NATS server and track their delivery position. When Telegraf reconnects, the durable consumer resumes from where it left off — no replay, no message loss.
### Expected behavior
The NATS Consumer plugin should support configuring a durable consumer name and related JetStream consumer options (deliver policy, ack wait, max deliver, filter subjects). When a durable name is set, the consumer should survive Telegraf restarts and resume processing from the last acknowledged position. On shutdown, the subscription should be drained (not unsubscribed) to preserve the consumer on the server.
### Actual behavior
The plugin only supports ephemeral push consumers. There is no way to configure a durable consumer name, deliver policy, or other JetStream consumer options. On shutdown the subscription is unsubscribed, destroying the ephemeral consumer. Restarting Telegraf creates a fresh consumer with no knowledge of previously delivered messages.
### Additional info
A PR implementing this feature is available: #18720
Contributor guide
Research direction
Start with the inputs.nats_consumer plugin and review PR #18720, which is identified as implementing this feature. Done means durable consumer configuration supports the listed JetStream options, resumes after restart, and drains subscriptions on shutdown without destroying the server-side consumer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100