snowflakedb / snowflakedb/snowflake-connector-python

SNOW-846438: Getting 0 rows from a TIMESTAMP_TZ column gives timezone-less pandas dtype

Open
#1,612 5 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

bug status-in_progress status-triage_done triaged
Dominant language
Python
Stars
730
Forks
574
Avg merge
5h 45m
Merged PRs (30d)
16

Description

Python version

Python 3.8.16 (default, Mar 1 2023, 21:19:10) [Clang 14.0.6 ]

Operating system and processor architecture

macOS-10.16-x86_64-i386-64bit

Installed packages
aiobotocore==2.5.0
aiohttp==3.8.4
aioitertools==0.11.0
aiosignal==1.3.1
appnope==0.1.3
asn1crypto==1.5.1
asttokens==2.2.1
async-timeout==4.0.2
attrs==23.1.0
backcall==0.2.0
bleach==6.0.0
boto3==1.26.148
botocore==1.29.153
cachetools==5.3.1
certifi==2023.5.7
cffi==1.15.1
charset-normalizer==2.1.1
cloudpickle==2.0.0
contourpy==1.0.7
cryptography==40.0.2
cycler==0.11.0
db-dtypes==1.1.1
decorator==5.1.1
docutils==0.20.1
duckdb==0.8.0
exceptiongroup==1.1.1
execnet==1.9.0
executing==1.2.0
filelock==3.12.0
fonttools==4.39.4
frozenlist==1.3.3
fsspec==2023.6.0
google-api-core==2.11.0
google-auth==2.19.1
google-cloud-bigquery==3.11.0
google-cloud-core==2.3.2
google-crc32c==1.5.0
google-resumable-media==2.5.0
googleapis-common-protos==1.59.0
grpcio==1.54.2
grpcio-status==1.54.2
idna==3.4
importlib-metadata==6.6.0
importlib-resources==5.12.0
iniconfig==2.0.0
ipdb==0.13.13
ipython==8.12.2
jaraco.classes==3.2.3
jedi==0.18.2
jmespath==1.0.1
joblib==1.2.0
keyring==23.13.1
kiwisolver==1.4.4
markdown-it-py==2.2.0
matplotlib==3.7.1
matplotlib-inline==0.1.6
mdurl==0.1.2
-e git+ssh://git@github.com/mvashishtha/modin.git@eeb410c87aa657d556d3cf2403195bdc76cc7193#egg=modin
more-itertools==9.1.0
multidict==6.0.4
Nuitka==1.6.1
numpy==1.24.3
ordered-set==4.1.0
oscrypto==1.3.0
packaging==23.1
pandas==1.5.3
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.5.0
pkginfo==1.9.6
pluggy==1.0.0
prompt-toolkit==3.0.38
proto-plus==1.22.2
protobuf==4.23.2
psutil==5.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==10.0.1
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
pycryptodomex==3.18.0
Pygments==2.15.1
PyJWT==2.7.0
pyOpenSSL==23.2.0
pyparsing==3.0.9
pytest==7.3.1
pytest-mock==3.10.0
pytest-xdist==3.3.1
python-dateutil==2.8.2
pytz==2023.3
readme-renderer==37.3
requests==2.31.0
requests-toolbelt==1.0.0
rfc3986==2.0.0
rich==13.4.1
rsa==4.9
s3fs==0.4.2
s3transfer==0.6.1
schedule==1.2.0
scikit-learn==1.2.2
scipy==1.10.1
six==1.16.0
snowflake-connector-python==3.0.3
stack-data==0.6.2
threadpoolctl==3.1.0
tomli==2.0.1
traitlets==5.9.0
twine==4.0.2
typing_extensions==4.6.3
urllib3==1.26.16
wcwidth==0.2.6
webencodings==0.5.1
wrapt==1.15.0
yarl==1.9.2
zipp==3.15.0
zstandard==0.21.0
What did you do?
# con is a snowflake.connector.connection.SnowflakeConnection
cursor = con.cursor()
# first select all the rows
cursor.execute("""SELECT '2021-01-01 00:00:00 +0000'::timestamp_tz AS TIME""")
# i get dtypes with timezone: TIME    datetime64[ns, America/Los_Angeles]
print(cursor.fetch_pandas_all().dtypes)
# now filter out the only record.
cursor.execute("""SELECT * FROM (SELECT '2021-01-01 00:00:00 +0000'::timestamp_tz AS TIME) WHERE TIME IS NULL""")
print(cursor.fetch_pandas_all().dtypes)
# Now I lose the timezone. TIME    datetime64[ns]
What did you expect to see?

This experience doesn't match the pandas experience, where filtering out all the records from a column should preserve the column's dtype. I want to get datetime64[ns, America/Los_Angeles] (my session time zone) even when my query returns non records.

Can you set logging to DEBUG and collect the logs?

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.