NatLabRockies / NatLabRockies/GridAnalysisToolkit

Explore replacing pyarrow with arro3 for a lighter Arrow dependency

Open
#30 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement performance
Dominant language
Python
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

pyarrow is a large, C++-backed dependency (contributed heavily to the slow-first-import findings in #22, and now sits in the plexos/server/client extras per #23's packaging split). arro3 is a much smaller, Rust-backed minimal Arrow implementation — worth evaluating as a lighter alternative for GAT's actual usage.

Current pyarrow usage (verified, not guessed)

Exactly three call sites across the whole codebase:

  1. gat/server/routes.py — Arrow IPC serialization for HTTP responses: pa.Table.from_pandas(df), pa.BufferOutputStream(), pa.ipc.new_stream(sink, table.schema).
  2. gat/client/connection.py — the corresponding deserialization: pa.ipc.open_stream(resp.content).
  3. gat/backends/duckdb_backend.pydf.to_parquet(..., compression="zstd"), which goes through pandas' own parquet engine dispatch, not a direct pyarrow API call.

Two different replacement paths

  • (1) and (2), the Arrow IPC transport layer — straightforwardly arro3's core strength (Arrow C Data Interface + IPC serialization in a much smaller package). Likely a clean swap.
  • (3), parquet writing — harder. pandas.to_parquet(engine=...) is hard-wired to recognize pyarrow or fastparquet by name; arro3 isn't (currently) a registered pandas parquet engine, so this call site can't just swap the import. Two options: (a) keep pyarrow scoped narrowly to just this one parquet-writing path, or (b) — possibly the better fix — write parquet via DuckDB's own native writer (COPY ... TO 'file.parquet' / relation.write_parquet(...)) instead of routing through pandas at all, which sidesteps needing any parquet-engine package here since duckdb is already a hard dependency everywhere _write_parquet is called.

Scope

Exploration only — needs confirming arro3's current API maturity/stability for (1)/(2), and deciding on (3)'s approach, before any implementation PR. If (3) moves to DuckDB-native writing, pyarrow could potentially be dropped from plexos/server entirely and arro3 (much smaller) used only where client/server actually need Arrow IPC.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the three stated call sites: gat/server/routes.py, gat/client/connection.py, and gat/backends/duckdb_backend.py, along with the current extras split from #23. Evaluate arro3's IPC API for the server and client paths, then determine whether the parquet path should retain pyarrow or use DuckDB-native writing. Done means documenting the compatibility findings and choosing a replacement scope before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, build-system
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.