tamnd / tamnd/firepanda

M3: read_parquet is not in the extension, because dlopen does not survive the shared library build

Open
#211 0 comments 0 reactions 0 assignees View on GitHub
area/python packaging
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

`read_parquet` cannot go in the extension yet, and the reason is `dlopen`.

The first `DataFrame` binding landed with `read_csv` and without `read_parquet`, which was not the plan. This issue records why and what it would take.

## What happens

Importing `firepanda.io` into the extension pulls in the Parquet reader, which pulls in the DuckDB loader, which calls `dlopen` through `external_call`. The import chain is `firepanda/io/__init__.mojo:62` to `firepanda/io/parquet.mojo:49` to `firepanda/io/duckdb.mojo:513`. Building that into a shared library fails twice over, first with an existing function with conflicting signature against `std/ffi/__init__.mojo:1249` and then with a failure to legalize operation `pop.external_call`. Neither failure happens when the same code is built as an executable, which is why the test suite has never seen it.

The workaround in place is to import `firepanda.io.read` directly rather than `firepanda.io`, which reaches `read_csv` without going past the Parquet reader. That is a workaround and not a fix. It will stop working the moment anything else in the extension wants something from `firepanda.io`.

## Why it is not just a build problem

Even with the build fixed there is a second question, and it is the more important one. `docs/specs/12-the-python-front-door-measured.md` section 2 says the wheel carries everything it needs and nothing points back at the machine that built it, and `python/tests/test_extension.py` holds that claim to a child interpreter with the environment stripped away. A Parquet reader that loads `libduckdb` at runtime is a hole in that claim. Either the wheel grows a DuckDB dependency, which is a large one and changes what firepanda is to install, or `read_parquet` raises a clear error when the library is absent and the wheel stays self contained with a documented gap in it.

That is a decision rather than a bug, and it should be made before the build is fixed, because the two answers want different things from the build.

## What done looks like

- The shared library build survives an import of `firepanda.io`, or the Parquet reader stops being on that path.
- `firepanda.read_parquet` exists on the Python side, or there is a written reason in the specification why it does not.
- If it exists and DuckDB is not vendored, calling it without `libduckdb` present raises something that says so, and there is a test for that.
- `python/tests/test_extension.py` still passes with its environment stripped, which is the thing this must not quietly break.

Refs #5, #4.

Contributor guide

Open the contributing guide

Research direction

Read the import chain from firepanda/io/__init__.mojo:62 through firepanda/io/parquet.mojo:49 and firepanda/io/duckdb.mojo:513, then review section 2 of docs/specs/12-the-python-front-door-measured.md. Use python/tests/test_extension.py with its stripped environment to evaluate the dependency decision and shared-library behavior. Done means the stated read_parquet policy is implemented or documented, with the missing-libduckdb case tested and the extension test still passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.