M3: read_parquet is not in the extension, because dlopen does not survive the shared library build
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- Avg merge
- 1h 31m
- Merged PRs (30d)
- 640
Description
`read_parquet` cannot go in the extension yet, and the reason is `dlopen`.
The first `DataFrame` binding landed with `read_csv` and without `read_parquet`, which was not the plan. This issue records why and what it would take.
## What happens
Importing `firepanda.io` into the extension pulls in the Parquet reader, which pulls in the DuckDB loader, which calls `dlopen` through `external_call`. The import chain is `firepanda/io/__init__.mojo:62` to `firepanda/io/parquet.mojo:49` to `firepanda/io/duckdb.mojo:513`. Building that into a shared library fails twice over, first with an existing function with conflicting signature against `std/ffi/__init__.mojo:1249` and then with a failure to legalize operation `pop.external_call`. Neither failure happens when the same code is built as an executable, which is why the test suite has never seen it.
The workaround in place is to import `firepanda.io.read` directly rather than `firepanda.io`, which reaches `read_csv` without going past the Parquet reader. That is a workaround and not a fix. It will stop working the moment anything else in the extension wants something from `firepanda.io`.
## Why it is not just a build problem
Even with the build fixed there is a second question, and it is the more important one. `docs/specs/12-the-python-front-door-measured.md` section 2 says the wheel carries everything it needs and nothing points back at the machine that built it, and `python/tests/test_extension.py` holds that claim to a child interpreter with the environment stripped away. A Parquet reader that loads `libduckdb` at runtime is a hole in that claim. Either the wheel grows a DuckDB dependency, which is a large one and changes what firepanda is to install, or `read_parquet` raises a clear error when the library is absent and the wheel stays self contained with a documented gap in it.
That is a decision rather than a bug, and it should be made before the build is fixed, because the two answers want different things from the build.
## What done looks like
- The shared library build survives an import of `firepanda.io`, or the Parquet reader stops being on that path.
- `firepanda.read_parquet` exists on the Python side, or there is a written reason in the specification why it does not.
- If it exists and DuckDB is not vendored, calling it without `libduckdb` present raises something that says so, and there is a test for that.
- `python/tests/test_extension.py` still passes with its environment stripped, which is the thing this must not quietly break.
Refs #5, #4.
Contributor guide
Research direction
Read the import chain from firepanda/io/__init__.mojo:62 through firepanda/io/parquet.mojo:49 and firepanda/io/duckdb.mojo:513, then review section 2 of docs/specs/12-the-python-front-door-measured.md. Use python/tests/test_extension.py with its stripped environment to evaluate the dependency decision and shared-library behavior. Done means the stated read_parquet policy is implemented or documented, with the missing-libduckdb case tested and the extension test still passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100