snowflakedb / snowflakedb/snowflake-connector-python

SNOW-715862: fetch_pandas_all for large timestamp get out of bound timestmap error

Open
#1,378 18 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature status-triage_done triaged
Dominant language
Python
Stars
730
Forks
574
Avg merge
5h 45m
Merged PRs (30d)
16

Description

Please answer these questions before submitting your issue. Thanks!

  1. What version of Python are you using?
    Python 3.9

  2. What operating system and processor architecture are you using?

Ubuntu

  1. What are the component versions in the environment (pip freeze)?

snowflake-connector-python==2.8.3

  1. What did you do?
    cur.execute("select TO_TIMESTAMP('9999-01-01T00:00:59Z') as d1")
    df = cur.fetch_pandas_all()
    will throw following exception:
File \"/usr/local/lib/python3.9/site-packages/pyarrow/pandas_compat.py\", line 1153, in _table_to_blocks\n    result = pa.lib.table_to_blocks(options, block_table, categories,\n  File \"pyarrow/table.pxi\", line 2602, in pyarrow.lib.table_to_blocks\n  File \"pyarrow/error.pxi\", line 100, in pyarrow.lib.check_status\npyarrow.lib.ArrowInvalid: Casting from timestamp[us] to timestamp[ns] would result in out of bounds timestamp: 253402300799000000"}
  1. What did you expect to see?
    fetch_pands_all() should allow all valid timestamp to loaded, for out of bound timestamp, allow a way to return NaT or max Timestamp value

  2. Can you set logging to DEBUG and collect the logs?

    import logging
    import os
    
    for logger_name in ('snowflake.connector',):
        logger = logging.getLogger(logger_name)
        logger.setLevel(logging.DEBUG)
        ch = logging.StreamHandler()
        ch.setLevel(logging.DEBUG)
        ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
        logger.addHandler(ch)
    

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the TO_TIMESTAMP query and fetch_pandas_all() call with the listed Python, connector, Ubuntu, and PyArrow environment. Trace the pandas-fetch conversion path that turns the large timestamp into a pandas value, then define and test the expected handling for valid out-of-bounds timestamps, such as NaT or the maximum timestamp.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.