vectordotdev / vectordotdev/vector
Support `orc` columnar encoding format in `aws_s3` sink
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
The ORC format is a columnar format that enables faster and more efficient data access schemes such as column selection and indexing. We used this within the Timber ingestion pipeline and saw a roughly 90% performance increase over standard text or CSV files. Our tests, as well as other benchmarks show ORC to have slightly better performance over parquet.
Best Practices
There are a few best practices for this format in the context of logs that were rigorously tested as part of the Timber pipeline development:
- A good default for the index step size is 10,000 records.
- Compressing ORC files with LZ4 (or gzip) has significant performance and size improvements since S3 data must be transferred over a network before being processed.
- While timestamp sorted data takes better advantage of ORC's indexes, we found that it was not necessary for the logging use case. This is due to the fact that log data is typically in-order as it is received. Strict ordering is not required to build useful indexes.
- Don't worry about bloom filters for this first version.
Implementation
The implementation for this feature is going to be interesting. I couldn't find a Rust crate for writing ORC data, and the official ORC library is written in Java, so I think we have 3 options:
- Attempt to follow the ORC spec and write a very rudimentary format that only supports the above requirements (does not concern itself with bloom filters and other irrelevant intricacies).
- Contract someone to do the above.
- Pass on supporting this feature.
I will say that recreating this in Rust would be an interesting performance comparison given that this is used within high volume logging pipelines.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing aws_s3 sink and reviewing the ORC specification and the listed logging best practices. Decide whether a Rust writer or another approach is viable, then define support for the stated index and compression requirements; completion would require an agreed implementation and validation against the sink's performance and output expectations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100