vectordotdev / vectordotdev/vector

A schema transform that can build and guarantee consistent types

Open
#2,227 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

domain: schemas domain: transforms meta: feedback meta: idea needs: approval type: help
Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

A very common problem with any sink that requires a schema is maintaining consistent types, especially in the logging use case. For example, elasticsearch will reject data if the types are not consistent. One way of solving this is to front think sink with a static schema, but this requires the operator to know about all fields and types upfront. This is usually impossible for logging pipelines since upstream data is constantly changing. A couple of ideas:

  1. Vector could dynamically build a schema to ensure consistent types. This, of course, gets very complicated when you have more than one Vector instance running.
  2. Vector could build a schema in a remote source, such as AWS Glue, AWS Dynamo, or any database that offers locking. This, again, can be somewhat complicated.

It's possible that the best solution is to do nothing and defer this to the downstream storage. Elasticsearch offers both a ignore_malformed and corece settings for exactly this.

Related #165.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No implementation files, tests, or entry points are named. First clarify whether Vector should build a local schema, coordinate through a remote store, or defer consistency to Elasticsearch; done requires a decided approach and a defined guarantee for changing fields and multiple Vector instances.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, elasticsearch, rust
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.