vectordotdev / vectordotdev/vector

Length-delimited framing actually emitting big-endian length, not little-endian

Open Beginner friendly
#19,699 0 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

domain: codecs
Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

A note for the community
  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem

The docs for framing in sinks say:

length_delimited

Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.

Where in fact this framing emits Big Endian-encoded integer.
Documentation needs to be fixed.

See here, the tokio_util::codec::LengthDelimitedCodec codec is used:
https://github.com/vectordotdev/vector/blob/7cf2f009dbd9be4177dfbce7950cd82d57f93448/lib/codecs/src/encoding/framing/length_delimited.rs#L25

And this codec emits big-endian: https://docs.rs/tokio-util/latest/tokio_util/codec/length_delimited/#getting-started

The returned transport implements Sink + Stream for BytesMut. It encodes the frame with a big-endian u32 header denoting the frame payload length:

I've confirmed it when I was writing a server to receive Vector events.

Configuration
socket:
    type: socket
    inputs:
    - parse_logs
    
    mode: unix
    path: /tmp/foo.socket

    encoding:
      codec: text
    
    framing:
      method: length_delimited
    
    healthcheck:
      enabled: false
Version

0.35

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Search the repository documentation for the length_delimited framing text under sinks, then compare it with the referenced lib/codecs/src/encoding/framing/length_delimited.rs implementation and the tokio-util codec documentation. Done means the documented byte order matches the big-endian framing actually emitted by LengthDelimitedCodec.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.