apache / apache/datafusion

Question: does datafusion-cli being tested on all files in apache/parquet-testing?

Open
#17,828 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

I found a parquet file that datafusion-cli failed to read and opened a bug in:
- https://github.com/apache/arrow-rs/issues/8495

and added the file in:
- https://github.com/apache/parquet-testing/pull/96

I can't find it but does the CI run basic sanity on all the files in https://github.com/apache/parquet-testing data directory (https://github.com/apache/parquet-testing/tree/a3d96a65e11e2bbca7d22a894e8313ede90a33a3/data)

like simple:
```bash
find parquet-testing/data/ -name "*.parquet" -type f | while read file; do
echo "Processing: $file"
datafusion-cli --command "select * from '$file'"
done
```

for sanity test

Contributor guide

Open the contributing guide

Research direction

Start by checking the existing CI and datafusion-cli entry points for any use of the apache/parquet-testing data directory. Compare the current coverage with the proposed find loop and determine whether a sanity run over all .parquet files is practical. Done means the CI behavior and failure reporting are explicitly established.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.