vectordotdev / vectordotdev/vector

Support `orc` columnar encoding format in `aws_s3` sink

Open
#1,373 4 comments 10 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

domain: codecs have: nice needs: approval needs: requirements sink: aws_s3
Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

The ORC format is a columnar format that enables faster and more efficient data access schemes such as column selection and indexing. We used this within the Timber ingestion pipeline and saw a roughly 90% performance increase over standard text or CSV files. Our tests, as well as other benchmarks show ORC to have slightly better performance over parquet.

Best Practices

There are a few best practices for this format in the context of logs that were rigorously tested as part of the Timber pipeline development:

  • A good default for the index step size is 10,000 records.
  • Compressing ORC files with LZ4 (or gzip) has significant performance and size improvements since S3 data must be transferred over a network before being processed.
  • While timestamp sorted data takes better advantage of ORC's indexes, we found that it was not necessary for the logging use case. This is due to the fact that log data is typically in-order as it is received. Strict ordering is not required to build useful indexes.
  • Don't worry about bloom filters for this first version.

Implementation

The implementation for this feature is going to be interesting. I couldn't find a Rust crate for writing ORC data, and the official ORC library is written in Java, so I think we have 3 options:

  1. Attempt to follow the ORC spec and write a very rudimentary format that only supports the above requirements (does not concern itself with bloom filters and other irrelevant intricacies).
  2. Contract someone to do the above.
  3. Pass on supporting this feature.

I will say that recreating this in Rust would be an interesting performance comparison given that this is used within high volume logging pipelines.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing aws_s3 sink and reviewing the ORC specification and the listed logging best practices. Decide whether a Rust writer or another approach is viable, then define support for the stated index and compression requirements; completion would require an agreed implementation and validation against the sink's performance and output expectations.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.