vectordotdev / vectordotdev/vector
Tremor sink
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
Hi! Greetings from the tremor team :)
today tremor and vector can talk over TCP with a configuration as mentioned below (excuse the yaml and toml salad)
onramp:
- id: vector
type: tcp
codec: string
preprocessors:
- lines
config:
host: 127.0.0.1
port: 5000
binding:
- id: vector
links:
'/onramp/vector/{instance}/out': [ '/pipeline/main/{instance}/in' ]
'/pipeline/main/{instance}/out': [ '/offramp/system::stdout/{instance}/in' ]
pipeline:
- id: main
interface:
inputs:
- in
outputs:
- out
links:
in: [ out ]
mapping:
/binding/vector/01:
instance: "01"
and
# Input data.
[sources.in]
type = "stdin"
# Output data
[sinks.out]
type = "socket"
inputs = ["in"]
address = "127.0.0.1:5000"
mode = "tcp"
encoding = "text"
The socket sink in text mode uses a new line delimiter which tremors can accept via the line preprocessor - that is awesome, huzza standards! But it might be worth considering adding a tremor specific sink to use size prefixed encoding (or size delimited encoding, depending on what you want to call it). Tremor currently supports this via the size-prefixed preprocessor that decodes the first 64bit of every message in network byte order (big-endian) to determine the number of bytes the message contains.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by reviewing the existing socket sink and its TCP text-mode framing, then compare it with Tremor's size-prefixed format: completion would require a defined Tremor-specific sink or encoding path that emits each message with its big-endian 64-bit size prefix and coverage for that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100