Vector loads entire disk buffer on start up and could potentially cause OOM

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

We are running vector with systemd. We get vector crashlooping due to oom occasionally and it's becoming more frequent recently.

I think the main cause of oom is the s3 sink's disk buffer builds up over time and whenever we restart vector, vector will load the entire buffer and consume substantially more memory and trigger oom.

The root cause might be some issue with the disk buffer which causes the buffer size increasing non-stop. We see increased number of these logs around the time oom starts

2026-08-05T13:02:29.184458Z ERROR sink{component_kind="sink" component_id=sink-s3 component_type=aws_s3}: vector_buffers::internal_events: Error encountered during buffer read. error=The reader detected that a data file contains a partially-written record. error_code="partial_write" error_type="reader_failed" stage="processing"

Configuration

sink-s3:
        type: aws_s3
        inputs:
            - ...
        acl: bucket-owner-full-control
        bucket: log-archive-bucket
        filename_append_uuid: false
        healthcheck: false
        key_prefix: key/prefix
        region: ...
        encoding:
            codec: json
        buffer:
            max_size: 2147483648
            type: disk
            when_full: drop_newest
        compression: zstd
        storage_class: INTELLIGENT_TIERING

Version

0.57

Debug Output

Example Data

No response

Additional Context

No response

References

No response

Start by tracing the disk buffer startup read path and the handling of the reported partial_write error, using the supplied systemd configuration to reproduce the behavior. Done should mean restarting with a large disk buffer no longer loads the entire buffer into memory or causes an OOM crash, with the buffer continuing to handle partial records safely.