influxdata / influxdata/telegraf

feat(inputs.nats_consumer): add durable consumer support for JetStream

Open
#18,734 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

When running Telegraf with the NATS Consumer input plugin against JetStream streams, restarting Telegraf causes the ephemeral push consumer to be destroyed. On restart, a new ephemeral consumer is created and — depending on the deliver policy — messages are either replayed from the beginning of the stream or missed entirely. This makes Telegraf unsuitable for production JetStream workloads where at-least-once delivery and restart resilience are required.

Durable consumers persist on the NATS server and track their delivery position. When Telegraf reconnects, the durable consumer resumes from where it left off — no replay, no message loss.

### Expected behavior

The NATS Consumer plugin should support configuring a durable consumer name and related JetStream consumer options (deliver policy, ack wait, max deliver, filter subjects). When a durable name is set, the consumer should survive Telegraf restarts and resume processing from the last acknowledged position. On shutdown, the subscription should be drained (not unsubscribed) to preserve the consumer on the server.

### Actual behavior

The plugin only supports ephemeral push consumers. There is no way to configure a durable consumer name, deliver policy, or other JetStream consumer options. On shutdown the subscription is unsubscribed, destroying the ephemeral consumer. Restarting Telegraf creates a fresh consumer with no knowledge of previously delivered messages.

### Additional info

A PR implementing this feature is available: #18720

Contributor guide

Open the contributing guide

Research direction

Start with the inputs.nats_consumer plugin and review PR #18720, which is identified as implementing this feature. Done means durable consumer configuration supports the listed JetStream options, resumes after restart, and drains subscriptions on shutdown without destroying the server-side consumer.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.