snowflakedb / snowflakedb/snowflake-connector-python
SNOW-747083: Adding write_pandas the option to write json to snowflake
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 730
- Forks
- 574
- Avg merge
- 5h 45m
- Merged PRs (30d)
- 16
Description
What is the current behavior?
The current behavior is that I have a dataframe where one of the columns is json.
When I defined the table I made sure to set this column to be variant.
For some reason, the write_pandas write the data as string and not as json hence I can't query the data inside this column unless parsing it use parse json in snowflake.
What is the desired behavior?
Be able to save json column from dataframe using write_pandas.
How would this improve snowflake-connector-python?
It is something we all need.. maybe it is possible and I am missing anything, but from writing the dataframe to parquet and from there to snowflake, the json is lost and I get a string.
References, Other Background
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the write_pandas entry point and its dataframe-to-Parquet path, focusing on how JSON-valued columns are serialized for Snowflake VARIANT columns. Done means a dataframe JSON column is stored as queryable JSON rather than a string requiring PARSE_JSON; add or update coverage for this behavior if the relevant tests are found.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100