vectordotdev / vectordotdev/vector
A schema transform that can build and guarantee consistent types
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
A very common problem with any sink that requires a schema is maintaining consistent types, especially in the logging use case. For example, elasticsearch will reject data if the types are not consistent. One way of solving this is to front think sink with a static schema, but this requires the operator to know about all fields and types upfront. This is usually impossible for logging pipelines since upstream data is constantly changing. A couple of ideas:
- Vector could dynamically build a schema to ensure consistent types. This, of course, gets very complicated when you have more than one Vector instance running.
- Vector could build a schema in a remote source, such as AWS Glue, AWS Dynamo, or any database that offers locking. This, again, can be somewhat complicated.
It's possible that the best solution is to do nothing and defer this to the downstream storage. Elasticsearch offers both a ignore_malformed and corece settings for exactly this.
Related #165.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No implementation files, tests, or entry points are named. First clarify whether Vector should build a local schema, coordinate through a remote store, or defer consistency to Elasticsearch; done requires a decided approach and a defined guarantee for changing fields and multiple Vector instances.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, elasticsearch, rust
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100