apache / apache/arrow-adbc

Better Snowflake support for mapping snowflake Ints and Floats to Arrow/Pandas/Polars

Open
#3,008 2 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
C#
Stars
627
Forks
217
Avg merge
17h
Merged PRs (30d)
57

Description

### What feature or improvement would you like to see?

https://docs.snowflake.com/en/sql-reference/data-types-numeric

According to their own documentation..

![Image](https://github.com/user-attachments/assets/f8d7d54f-2f2e-4470-b3a9-73d308f6d893)

But it doesn't look like ADBC is using this info and sizing everything to the internal decimal(38,0) type Snowflake uses.

In reality Snowflake doesn't use a decimal(38,0) either. It finds the largest INT value in a micropartition and sizes that entire micropartiion Int column to 128, 64, 32, 16 or 8 bit integers..

Shouldn't the ADBC Driver look at the metadata and if a column is defined as a TinyInt it should allocate a PyArrow Int8 instead of a Int128.. This would save a lot of memory on the client side..

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.