influxdata / influxdata/influxdb

[2.X] Implement Scalable Multi-Tenant High-Availability Pattern for InfluxDB 2.x OSS

Open
#27,269 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Use case:
Currently, InfluxDB 2.x OSS users facing high-growth or multi-tenant SaaS requirements hit a "scale wall." Because the OSS version is natively single-node, users are forced to choose between expensive Enterprise licenses or manual, fragile sharding logic. This proposal provides a standardized Kubernetes blueprint to achieve horizontal ingest scaling and storage redundancy using only OSS components.

Proposal:
Introduce a "Shared-Nothing" Cluster pattern for Kubernetes deployments. This feature-set leverages Telegraf as a dynamic path-based router and the Edge Data Replication API to decouple compute-heavy ingestion from long-term storage. It allows InfluxDB OSS to scale horizontally at the ingest tier while maintaining a mirrored, high-availability storage "Hub" for multiple tenants.

Current behaviour:
The standard deployment is a single StatefulSet.

  • No Redundancy: A pod crash or node maintenance results in total downtime.

  • No Multi-Tenancy: All tenants share a single compute boundary; a "noisy neighbor" in one organization can crash the pod for all others.

  • Vertical Scaling Only: Users must increase CPU/RAM on a single pod indefinitely.

Desired behaviour:
A distributed architecture where:

  • Ingress: A Telegraf gateway captures tenant names from the URL path (e.g., /tenant-a/v2/write) and routes data to specific buckets.

  • Ingest Tier (Auto-Scaling): A stateless-like tier of InfluxDB pods that handles the "heat" of parsing and compression, governed by a Horizontal Pod Autoscaler (HPA).

  • Edge Replication: Automated background streaming from Ingest pods to a redundant Hub Tier.

  • Storage Hubs: A mirrored pair (hub-0, hub-1) acting as the "Source of Truth," allowing for zero-downtime upgrades and persistent data safety.

Alternatives considered:

  • InfluxDB 3.0 (IOx): While powerful, the transition path for existing v2.x Flux users is non-trivial, and the resource requirements for the 3.0 engine are significantly higher for small-to-medium SaaS startups.

  • Manual Sidecars: Users currently script their own replication logic, which is error-prone and lacks the standardized monitoring (queue tracking, lag metrics) provided by a formal Helm/Kubernetes pattern.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points. Start by mapping the existing single StatefulSet deployment and the proposed Telegraf, HPA, Edge Data Replication, and mirrored hub components; done would require an agreed Kubernetes/Helm blueprint covering tenant routing, replication, scaling, redundancy, and monitoring.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
databases, distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.