Buffer-full as a distinct DLQ trigger, with overflow to object storage

Problem

We run Vector as a regional aggregator behind edge forwarders with their own (smaller) local buffers. When a downstream sink is down long enough for its buffer to fill, backpressure propagates to the source, and the edge forwarders upstream start dropping events once their own buffers fill too.

A bigger disk buffer doesn't solve this so much as move the wall. It's bounded by the volume it sits on, it's per-instance rather than pooled, and it goes away with the instance, so surviving a multi-hour downstream outage means provisioning every aggregator for the worst outage you expect, and still losing the backlog if you lose the node. The useful comparison isn't buffer size but buffer fill time against realistic outage duration: at our volume a fully-provisioned local buffer absorbs minutes, while the outages it needs to survive run to hours. That gap doesn't close by adding disk. Object storage is elastic and outlives the process, which is why the escape valve wants to point there.

We've built a version of this by hand: a second Vector service whose only job is writing to S3, an L7 proxy in front of both, and an object-notification-triggered replay back through the primary pipeline once the downstream recovers. It works, but it's a lot of machinery for behavior that's conceptually just "buffer overflowed, write it somewhere durable, drain it later."

Today, failover outside Vector:

  forwarders ──→ [Vector: normalize] ──→ proxy ──healthy──→ [Vector: primary sinks] ──→ downstream
                                           │
                                           └─degraded──→ [Vector: S3-only sink] ──→ S3
                                                                                     │
                                       object event ──→ queue ──→ replay source ─────┘

With buffer.overflow_sink:

  forwarders ──→ [Vector: sink] ──────────────────────────────────────────────────→ downstream
                        └── buffer full ──→ S3
                                             │
                         object event ──→ queue ──→ replay source ──→ (back in)

Two things make that shape harder than it looks, and both are arguments for doing this inside Vector:

• A full buffer isn't observable outside the process. When a sink's buffer fills under when_full: block, backpressure reaches the vector source and its send_batch pends, so no response is written and the call hangs rather than failing. Bounding the downstream sink's retry_attempts converts that into a delivery error, but that's a different signal at a real cost: it reports "delivery failed," never "buffer full," and it means giving up indefinite retry. Either way there's nothing an external proxy can route on that distinguishes a full buffer from a slow sink, which leaves a request timeout as the trigger.
• The overflow round trip is codec-bound. An aws_s3 sink has exactly one encoding.codec and an aws_s3 source exactly one decoding.codec, so the overflow path can only carry a single codec. Producing a single-codec stream means normalizing before the failover point, which is what forces the topology into two separate Vector services with a proxy between them.

Everything downstream of that first ambiguity is tuning, and there's a lot of it: by a wide margin the failover layer, not the durable write, has been the expensive part to build and the hard part to gain confidence in, and none of that effort is about not losing data. Overflowing from inside the sink has neither problem: the buffer knows its own state, and events reaching it are already through the pipeline's transforms.

Related:

• #1772: DLQ-on-sinks tracking issue, open since 2020
• #14708: "Handling Discarded Events" RFC, still an open draft. Stalled on design disagreement in 2022, with an implementation attempt proposed against it in March 2026.
• #24930: that implementation, adding a <sink_id>.dlq output port (Elasticsearch first). It triggers on non-retryable delivery failures from the downstream rather than on a full buffer, so it addresses a different half of #1772 than this does.
• https://github.com/vectordotdev/vector/discussions/22901: "Use alternative sink if primary is failing?" (Apr 2025), answered "not supported yet"

The gap: buffer-full is a distinct trigger

The prior art all fires on an error. #24930 routes events the downstream rejected; #14708 generalizes that to discarded events across components, and its Pain section lists transform processing, sink encoding failure, sink partitioning failure and failure to write to a disk buffer. That last one does reach into buffers, but it's an I/O error. A full buffer under when_full: block isn't an error at all, it's flow control working exactly as designed, which is why it falls outside a framework built around "events discarded due to errors outside the operator's control."

The events differ too, not just the trigger. #14708 wants discarded events captured "for diagnosis," which fits a handful of bad records. Overflow events aren't bad: they'd succeed on retry, there are as many of them as the outage is long, and the point is to replay every one rather than inspect a few. Nothing currently proposed covers that case.

There are two places it could live, and I think one of them is clearly right.

As an additional trigger on the emerging output-port work. If #24930's <sink_id>.dlq port lands, the graph, validation and topology-builder changes are already paid for, and buffer-full becomes another condition that feeds a port routed to a normal aws_s3 sink. One concept instead of two, and any destination rather than object storage only. The complication is that an overflow port has to be a side channel exempt from the very backpressure that triggered it, and has to be prevented from routing back into the stalled sink, which a general graph edge makes easy to get wrong.

As a buffer-local side channel. The shape below. It's harder to misconfigure because the destination is constrained and never participates in the topology, but it's a second mechanism covering adjacent ground, which is a real cost if the port work lands.

The buffer-local route is the one I'd build. The deadlock isn't a configuration edge case, it's the default result of pointing an overflow port at anything downstream of the sink that stalled, and a design you have to document your way around is worse than one that can't express the mistake. Constraining the destination gives up genericity nobody is asking for: #1772, the discussion linked above and this issue all want durable storage, not arbitrary rerouting.

Proposed shape: buffer.overflow_sink

• Not a buffer.type variant. memory/disk/disk_v2 all satisfy a read/write contract (dequeue back into the sink), whereas this is write-only by design, a side channel that activates only on overflow.
• Scoped to object storage sinks (aws_s3 / gcp_cloud_storage / azure_blob, selected via type the same way buffer.type discriminates buffer variants) rather than arbitrary sink configs. Object storage sinks share simple, uniform write semantics (durable, at-least-once, no bidirectional protocol). Arbitrary sink support reopens per-destination protocol/retry questions this should avoid.
• On the name: overflow_sink reuses "overflow" deliberately rather than avoiding a clash with when_full: overflow. The two describe the same relationship from either end: when_full: overflow is the trigger condition, overflow_sink is the destination it triggers into. Arguably the when_full value is then redundant, since the presence of the overflow_sink key already implies the behavior; either shape works and I don't have a strong view.
• If the overflow sink itself fails (object storage unreachable, credentials expired, throttling), the buffer falls back to whatever when_full behavior it had before, which is block in most configurations. Overflow is a best-effort escape valve on top of the existing contract, never a replacement for it, so a broken overflow destination degrades to today's behavior rather than to data loss.
• Replay is a separate, deliberate step, a normal source reading the objects back in rather than something Vector auto-drains. Auto-drain needs a "safe to resume" heuristic and its own backoff policy, and is architecturally novel (no sink today owns and manages a source internally). For what it's worth, decoupled replay is the part of our hand-rolled version that has needed the least attention: paced by end-to-end acks, it drains as capacity returns without ever overrunning a downstream that's still catching up.

sinks:
  primary:
    type: splunk_hec_logs
    buffer:
      type: disk
      when_full: overflow_sink
      overflow_sink:
        type: aws_s3
        bucket: my-overflow-bucket
        batch:
          max_bytes: 30000000
          timeout_secs: 5
        compression: gzip

Ack semantics

Events ack on write to overflow_sink, not on eventual delivery to the primary. Otherwise a source with acks enabled has no way to release an upstream forwarder, since replay timing is unbounded. This isn't new behavior: disk buffers already ack on persist-to-disk rather than on delivery, so this is the same rule one stage further down the chain.

Delivery through the overflow path is at-least-once and out of order relative to live traffic, since replayed events interleave with whatever is arriving at the time. That's the same tradeoff the existing workaround makes, and it's the right one for a "don't lose data" escape valve.

What overflow_sink can't do

It can't expose "unreplayed" state, which is a consumption-side fact, and the write side has no visibility into a separate, decoupled replay pipeline. What it can expose is standard write-side telemetry (objects/bytes written, last-write timestamp). Tracking whether the backlog is actually draining is the operator's responsibility on their own replay pipeline.

Implementation notes

Code references are to master at time of writing:

• Reusing aws_s3's SinkConfig::build() for the overflow destination is sound. Every sink implements that trait ((VectorSink, Healthcheck)), and the topology builder already calls it (sink.inner.build(cx)) for the primary sink, so building a second instance for overflow_sink isn't new territory.
• Stream branching is more work than it first looks. Today sink.buffer.build(...) produces one (tx, rx) pair, piped straight into a single sink.run(rx), with no existing branch point for a second consumer. The overflow signal (WhenFull) lives inside vector-buffers, which has no dependency on sink implementations today. Realistically this needs either restructuring sink-building to construct both sinks and split the stream (with correct finalizer/ack resolution per path), or a generic "overflow writer" hook in vector-buffers with the concrete sink injected from the topology builder. Neither is small, though neither touches topology or validation-graph logic, since a buffer-local overflow has no output to declare. On cost alone the comparison isn't one-sided, though: #24930 already writes those graph, validation and topology-builder changes, so if it lands, that cost is sunk and adding a trigger to an existing port is plausibly less work than a parallel mechanism in the buffer. The case for the buffer-local route is the deadlock argument above, not the diff size.

What I'm asking for

1. Has buffer-full as a trigger, as opposed to delivery rejection, been considered or rejected before? I couldn't find it discussed separately from #1772's general DLQ framing.
2. Buffer-local side channel, or a trigger on #24930's output port? I've argued for the first above.
3. Does it need an RFC, or is a decision here enough to start?

Worth naming the pattern: #1772 has been open since 2020, #14708 has been a draft since 2022, #24930 has sat unreviewed since March. None of them stalled because anyone disputed the problem, they stalled on design disagreement that never got a maintainer decision. A single-trigger, single-destination feature is the smallest thing that can break that, which is the whole reason this is scoped the way it is.

I'll write the RFC and submit the implementation. What I need is a decision on the shape.

Start by reading the buffer and sink entry points named in the issue: vector-buffers' WhenFull handling, sink.buffer.build(...), and sink.inner.build(cx). Compare the proposed buffer-local overflow side channel with the output-port approach in #24930, including stream branching and ack behavior. Done requires an agreed design and implementation path, not just a configuration change.