[C++] Parquet pushdown filtering fails if the filter expression uses numeric field references
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
We can refer to a field by name (e.g. `compute::field_ref("foo")`) or by index (e.g. `compute::field_ref(0)`).
The latter is not supported when doing parquet projection. A test can demonstrating this can be found here: https://github.com/westonpace/arrow/commit/2f92ed0764cf2e1388dac053aeb4e1b923c6872e
Copied here for posterity (this would go in the dataset fixture mixin):
```Java
void TestScanWithFieldPathFilter() {
auto i32 = field("i32", int32());
auto i64 = field("i64", int64());
this->opts_->dataset_schema = schema({i32, i64});
this->Project({"i64"});
// This should be the column i32
this->SetFilter(equal(field_ref(0), literal(0)));
auto expected_schema = schema({i64});
auto reader = this->GetRecordBatchReader(opts_->dataset_schema);
auto source = this->GetFileSource(reader.get());
auto fragment = this->MakeFragment(*source);
int64_t row_count = 0;
for (auto maybe_batch : PhysicalBatches(fragment)) {
ASSERT_OK_AND_ASSIGN(auto batch, maybe_batch);
row_count += batch->num_rows();
AssertSchemaEqual(*batch->schema(), *expected_schema,
/*check_metadata=*/false);
}
ASSERT_EQ(row_count, expected_rows());
}
```
I would expect this to work. Instead I get the error:
```
/home/pace/dev/arrow/cpp/src/arrow/dataset/test_util.h:840: Failure
Failed
'_error_or_value83.status()' failed with NotImplemented: Inferring column projection from FieldRef FieldRef.FieldPath(0)
/home/pace/dev/arrow/cpp/src/arrow/dataset/file_parquet.cc:262 ResolveOneFieldRef(manifest, ref, field_lookup, duplicate_fields, &columns_selection)
/home/pace/dev/arrow/cpp/src/arrow/dataset/file_parquet.cc:437 InferColumnProjection(*reader, *options)
/home/pace/dev/arrow/cpp/src/arrow/util/iterator.h:152 value_.status()
```
**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-15658) / @westonpace
#### Related issues:
- [[Python] Allow to create field reference to nested fields that use indices](https://github.com/apache/arrow/issues/31583) (blocks)
#### PRs and other links:
- [GitHub Pull Request #12664](https://github.com/apache/arrow/pull/12664)
**Note**: *This issue was originally created as [ARROW-15658](https://issues.apache.org/jira/browse/ARROW-15658). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with cpp/src/arrow/dataset/file_parquet.cc, especially ResolveOneFieldRef and InferColumnProjection, then inspect the dataset fixture test in cpp/src/arrow/dataset/test_util.h and the linked reproducer commit. Run the field-path filter test and confirm that projecting i64 while filtering on numeric field reference 0 succeeds with the expected schema and row count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100