apache / apache/arrow-rs

Fast non-validating reader mode to count records in Avro OCF files

Open
#9,613 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
A query like `SELECT COUNT(*) ...` on an Avro data source needs no data fields, only the number of rows in the partitioned data set.
With the Avro OCF format, this information can be obtained by decoding just the block frames, presuming that the data encoding is well-formed and the number of encoded records in each block matches the one stated in the block header.

**Describe the solution you'd like**
Add an option method to the reader builders that would make the reader bypass decompression and decoding of Avro record data. Instead, the decoder should only parse the OCF data blocks to sum the row counts, and produce record batches with no columns, but with the row counts and metadata corresponding to the file content. This method should not be used together with `with_reader_schema`.

The name of the method should give sufficient warning, e.g. `count_without_validation`.

**Describe alternatives you've considered**
This behavior could be enabled when the reader schema has no fields. However, since this could lead to invalid encoded data being accepted based on the block framing, it's preferable that an explicit option is used.

**Additional context**
#9608 concerns the behavior when the reader schema has no fields, but validation of Avro data is performed.

Contributor guide

Open the contributing guide

Research direction

Start with the Avro OCF reader builders and the data-block parsing path described in the issue, then compare the related behavior in issue #9608. Done means an explicit count_without_validation-style option sums block record counts without decompressing or decoding record data, returns empty-column batches with matching counts and metadata, and is not combined with with_reader_schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.