vectordotdev / vectordotdev/vector

Disk buffer startup fails permanently when seek hits decode error (InvalidProtobufPayload)

Open
#26,231 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

Problem

On restart, disk buffer (v2) initialization can fail permanently when seek_to_next_record() hits a record that fails protobuf decode:

error occurred when building buffer
failed to seek to position where reader left off
failed to decoded record: InvalidProtobufPayload

Checksum, deserialization, and partial_write errors during initialization seek are treated as recoverable "bad reads", but ReaderError::Decode was not — so startup fails until the buffer directory is deleted manually.

Expected behavior

Decode failures during initialization seek should follow the same recovery path as other bad reads (skip and continue), not fail topology build permanently.

Related reports
  • #18130, #19759, #25642 (same error family / seek-on-startup pattern)
Proposed fix

PR: https://github.com/vectordotdev/vector/pull/25691

Additional context

Tracked internally as Red Hat LOG-9468 (OpenShift Logging / AtLeastOnce disk buffer CrashLoop).

References

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing PR #25691, then trace disk buffer v2 initialization through seek_to_next_record() and ReaderError::Decode. Compare decode-error handling with checksum, deserialization, and partial_write recovery; done means initialization skips the invalid record and continues instead of failing topology startup.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.