vectordotdev / vectordotdev/vector

Tremor sink

Open
#1,944 0 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

domain: sinks needs: approval sink: new
Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

Hi! Greetings from the tremor team :)

today tremor and vector can talk over TCP with a configuration as mentioned below (excuse the yaml and toml salad)

onramp:
  - id: vector
    type: tcp
    codec: string
    preprocessors:
      - lines
    config:
      host: 127.0.0.1
      port: 5000

binding:
  - id: vector
    links:
      '/onramp/vector/{instance}/out': [ '/pipeline/main/{instance}/in' ]
      '/pipeline/main/{instance}/out': [ '/offramp/system::stdout/{instance}/in' ]

pipeline:
  - id: main
    interface:
      inputs:
        - in
      outputs:
        - out
    links:
      in: [ out ]


mapping:
  /binding/vector/01:
    instance: "01"

and

# Input data.
[sources.in]
  type = "stdin"

# Output data
[sinks.out]
  type = "socket"
  inputs   = ["in"]
  address = "127.0.0.1:5000"
  mode = "tcp" 
  encoding = "text"

The socket sink in text mode uses a new line delimiter which tremors can accept via the line preprocessor - that is awesome, huzza standards! But it might be worth considering adding a tremor specific sink to use size prefixed encoding (or size delimited encoding, depending on what you want to call it). Tremor currently supports this via the size-prefixed preprocessor that decodes the first 64bit of every message in network byte order (big-endian) to determine the number of bytes the message contains.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by reviewing the existing socket sink and its TCP text-mode framing, then compare it with Tremor's size-prefixed format: completion would require a defined Tremor-specific sink or encoding path that emits each message with its big-endian 64-bit size prefix and coverage for that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.