snowflakedb / snowflakedb/snowflake-connector-python

SNOW-2216201: Allow for passing pyarrow schema to write_pandas

Open
#2,416 1 comment 0 reactions 1 assignee View on GitHub

@sfc-gh-snow-drivers-warsaw-dl is already working on this.

Since Jul 23, 2025.

feature status-triage_done
Dominant language
Python
Stars
730
Forks
574
Avg merge
5h 45m
Merged PRs (30d)
16

Description

What is the current behavior?

I have a dataframe where one column holds python dicts. When I want to import it to snowflake using write_pandas it fails on memory (basically eats up all the memory and then fails on malloc).

To solve this I have to define a custom pyarrow schema. However I am unable to pass it as a kwarg to write_pandas because schema is already a named parameter.

Here's the complete flow:

pandas_tools.py
def write_pandas
chunk.to_parquet(chunk_path, compression=compression, **kwargs) # <-- if I pass schema=my_pyarrow_schema here, it works fine, but I cant because write_pandas already has a parameter named schema
What is the desired behavior?

Be able to pass pyarrow schema to write_pandas

How would this improve snowflake-connector-python?

Allow for non-standard dfs to be imported into SF.

References and other background

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.