NatLabRockies / NatLabRockies/GridAnalysisToolkit
Explore replacing pyarrow with arro3 for a lighter Arrow dependency
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
pyarrow is a large, C++-backed dependency (contributed heavily to the slow-first-import findings in #22, and now sits in the plexos/server/client extras per #23's packaging split). arro3 is a much smaller, Rust-backed minimal Arrow implementation — worth evaluating as a lighter alternative for GAT's actual usage.
Current pyarrow usage (verified, not guessed)
Exactly three call sites across the whole codebase:
gat/server/routes.py— Arrow IPC serialization for HTTP responses:pa.Table.from_pandas(df),pa.BufferOutputStream(),pa.ipc.new_stream(sink, table.schema).gat/client/connection.py— the corresponding deserialization:pa.ipc.open_stream(resp.content).gat/backends/duckdb_backend.py—df.to_parquet(..., compression="zstd"), which goes through pandas' own parquet engine dispatch, not a direct pyarrow API call.
Two different replacement paths
- (1) and (2), the Arrow IPC transport layer — straightforwardly arro3's core strength (Arrow C Data Interface + IPC serialization in a much smaller package). Likely a clean swap.
- (3), parquet writing — harder.
pandas.to_parquet(engine=...)is hard-wired to recognizepyarroworfastparquetby name; arro3 isn't (currently) a registered pandas parquet engine, so this call site can't just swap the import. Two options: (a) keep pyarrow scoped narrowly to just this one parquet-writing path, or (b) — possibly the better fix — write parquet via DuckDB's own native writer (COPY ... TO 'file.parquet'/relation.write_parquet(...)) instead of routing through pandas at all, which sidesteps needing any parquet-engine package here since duckdb is already a hard dependency everywhere_write_parquetis called.
Scope
Exploration only — needs confirming arro3's current API maturity/stability for (1)/(2), and deciding on (3)'s approach, before any implementation PR. If (3) moves to DuckDB-native writing, pyarrow could potentially be dropped from plexos/server entirely and arro3 (much smaller) used only where client/server actually need Arrow IPC.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the three stated call sites: gat/server/routes.py, gat/client/connection.py, and gat/backends/duckdb_backend.py, along with the current extras split from #23. Evaluate arro3's IPC API for the server and client paths, then determine whether the parquet path should retain pyarrow or use DuckDB-native writing. Done means documenting the compatibility findings and choosing a replacement scope before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, build-system
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100