fluent / fluent/fluent-bit

Support NATS Streaming Server (stan)

Open
#1,829 9 comments 3 reactions 1 assignee Claimed by @edsiper View on GitHub
enhancement
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

I would like to use NATS Streaming Server (aka stan -- https://github.com/nats-io/nats-streaming-server ) as my logging/analytics back-end. It uses a similar protocol to the existing NATS plugin but has a few additions/changes to support additional desirable features.

Reference:
A golang NATS Streaming Server client is available @ https://github.com/nats-io/stan.go

Steps:
Clone the existing nats plugin, replace connector code, review methods, arguments, configuration options.

Purpose:
NATS is lightweight (~30MB RAM) and ideal for streaming analytics data but the data is transient.

NATS Streaming Server is a lightweight addition (~10MB RAM) that allows for disk-backed, rewindable streams tracked per subscriber with at-least-once delivery. It runs on the same port/connection as NATS but uses a different client/protocol.

I am currently using a workaround of running a separate "replicator" service that copies NATS subjects to NATS Streams. I was able to use fluent-bit tail (throttled at ~1000 records/second) to import~5.6GB of nginx json access.log data (~5 million json records) with very little CPU/memory usage. The replicator: https://github.com/nats-io/nats-replicator

NATS Streaming Server is an ideal partner for event-sourcing Fluent Bit inputs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.