apache / apache/arrow

[C++] Parquet pushdown filtering fails if the filter expression uses numeric field references

Open
#20,114 6 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

We can refer to a field by name (e.g. `compute::field_ref("foo")`) or by index (e.g. `compute::field_ref(0)`).

The latter is not supported when doing parquet projection. A test can demonstrating this can be found here: https://github.com/westonpace/arrow/commit/2f92ed0764cf2e1388dac053aeb4e1b923c6872e

Copied here for posterity (this would go in the dataset fixture mixin):

```Java

void TestScanWithFieldPathFilter() {
auto i32 = field("i32", int32());
auto i64 = field("i64", int64());
this->opts_->dataset_schema = schema({i32, i64});
this->Project({"i64"});
// This should be the column i32
this->SetFilter(equal(field_ref(0), literal(0)));
auto expected_schema = schema({i64});
auto reader = this->GetRecordBatchReader(opts_->dataset_schema);
auto source = this->GetFileSource(reader.get());
auto fragment = this->MakeFragment(*source);

int64_t row_count = 0;

for (auto maybe_batch : PhysicalBatches(fragment)) {
ASSERT_OK_AND_ASSIGN(auto batch, maybe_batch);
row_count += batch->num_rows();
AssertSchemaEqual(*batch->schema(), *expected_schema,
/*check_metadata=*/false);
}

ASSERT_EQ(row_count, expected_rows());
}
```

I would expect this to work. Instead I get the error:

```

/home/pace/dev/arrow/cpp/src/arrow/dataset/test_util.h:840: Failure
Failed
'_error_or_value83.status()' failed with NotImplemented: Inferring column projection from FieldRef FieldRef.FieldPath(0)
/home/pace/dev/arrow/cpp/src/arrow/dataset/file_parquet.cc:262 ResolveOneFieldRef(manifest, ref, field_lookup, duplicate_fields, &columns_selection)
/home/pace/dev/arrow/cpp/src/arrow/dataset/file_parquet.cc:437 InferColumnProjection(*reader, *options)
/home/pace/dev/arrow/cpp/src/arrow/util/iterator.h:152 value_.status()
```

**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-15658) / @westonpace
#### Related issues:
- [[Python] Allow to create field reference to nested fields that use indices](https://github.com/apache/arrow/issues/31583) (blocks)
#### PRs and other links:
- [GitHub Pull Request #12664](https://github.com/apache/arrow/pull/12664)

**Note**: *This issue was originally created as [ARROW-15658](https://issues.apache.org/jira/browse/ARROW-15658). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with cpp/src/arrow/dataset/file_parquet.cc, especially ResolveOneFieldRef and InferColumnProjection, then inspect the dataset fixture test in cpp/src/arrow/dataset/test_util.h and the linked reproducer commit. Run the field-path filter test and confirm that projecting i64 while filtering on numeric field reference 0 succeeds with the expected schema and row count.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.