Support NATS Streaming Server (stan)
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
I would like to use NATS Streaming Server (aka stan -- https://github.com/nats-io/nats-streaming-server ) as my logging/analytics back-end. It uses a similar protocol to the existing NATS plugin but has a few additions/changes to support additional desirable features.
Reference:
A golang NATS Streaming Server client is available @ https://github.com/nats-io/stan.go
Steps:
Clone the existing nats plugin, replace connector code, review methods, arguments, configuration options.
Purpose:
NATS is lightweight (~30MB RAM) and ideal for streaming analytics data but the data is transient.
NATS Streaming Server is a lightweight addition (~10MB RAM) that allows for disk-backed, rewindable streams tracked per subscriber with at-least-once delivery. It runs on the same port/connection as NATS but uses a different client/protocol.
I am currently using a workaround of running a separate "replicator" service that copies NATS subjects to NATS Streams. I was able to use fluent-bit tail (throttled at ~1000 records/second) to import~5.6GB of nginx json access.log data (~5 million json records) with very little CPU/memory usage. The replicator: https://github.com/nats-io/nats-replicator
NATS Streaming Server is an ideal partner for event-sourcing Fluent Bit inputs.
Contributor guide
Assessment
This issue has not been assessed yet.