apache / apache/datafusion-python
Nothing in CI runs examples/*.py, and one of them is broken
- Dominant language
- Python
- Stars
- 604
- Forks
- 174
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 4
Description
Verified: the only `examples` paths in `.github/workflows/` are the five FFI crate directories and `examples/tpch`. **No top-level `examples/*.py` script is executed anywhere in CI.** Two consequences are already in the tree:
**`examples/csv-read-options.py` crashes.** It reads `data.csv` at lines 28, 39, 52, 72 and 78, and `data.csv.gz` at line 67. Neither file exists anywhere in the repository. This is not a script that runs quietly — it raises on line 28. Fix: have it write its own small CSV and gzip copy into a temporary directory at the top, which also makes it self-contained.
**Nine scripts print nothing at all.** `export.py`, `import.py`, `python-udaf.py`, `python-udf.py`, `query-pyarrow-data.py`, `sql-to-pandas.py`, `sql-using-python-udaf.py`, `sql-using-python-udf.py` and `substrait.py` contain zero `print` or `.show()` calls; they end in `assert`. They are tests wearing an example's filename. A script that prints nothing is indistinguishable to a reader from a script that does nothing. Fix is roughly two lines each: a terminal `df.show()` or `print(...)`, keeping the asserts.
**The root cause is the missing CI job**, and it is the part worth designing rather than just doing. A job running `examples/*.py` against the already-built wheel would have caught both, but it needs a skip matrix: `sql-parquet-s3.py` needs network and AWS credentials, and `sql-parquet.py`, `dataframe-parquet.py` and `sql-to-pandas.py` need an NYC taxi Parquet file the README tells you to download by hand. Worth agreeing the shape of that before writing it.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing .github/workflows/ and the entry points under examples/*.py, then run examples/csv-read-options.py to reproduce its missing-input failure. Define the wheel-based CI job and its skip matrix for the AWS and NYC taxi examples; done means the self-contained CSV example runs, the named silent scripts print output while retaining asserts, and eligible examples execute in CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100