Refactor ingestion, save and load functions
Open
@arienandalibi is already working on this.
Since Dec 1, 2025.
- Dominant language
- Rust
- Stars
- 642
- Forks
- 70
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 21
Description
After the merge of #2391 we will have the following functions:
- load_edges
- load_nodes
- load_edge_metadata
- load_node_metadata
These will handle any dataframe inputs including (tested):
- Pandas dataframes
- FireDucks(.pandas) dataframes
- Polars dataframes
- Arrow tables
- DuckDB (eg. DuckDBPyRelation obtained from running an SQL query)
In addition to this, we should modify the function to internally check if the user has provided a dataframe or a filepath as an argument and transparantly handle loading from this if possible. Through this we can remove the following functions as they would be rolled into the above:
- load_edge_props_from_pandas
- load_edge_props_from_parquet
- load_edges_from_pandas
- load_edges_from_parquet
- load_node_props_from_pandas
- load_node_props_from_parquet
- load_nodes_from_pandas
- load_nodes_from_parquet
To reduce confusion we will change the following:
- load_from_file -> load_graph
- save_to_file -> save_graph
- save_to_zip -> flag on save_graph (to be discussed)
Post v4 merge the following functions will be removed:
- from_parquet/to_parquet -- as we are removing the proto format and this will become the default format for save/load_graph
- load_cached/write_updates/cache -- we have the WAL now
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.