duckdb / duckdb/duckdb-python

read_csv() is not understanding some timestamp formats

Open
#442 2 comments 0 reactions 0 assignees View on GitHub
needs triage
Dominant language
Python
Stars
187
Forks
112
Avg merge
13h 29m
Merged PRs (30d)
17

Description

### What happens?

I am getting NaT (not a timestamp) for time stamps in format %Y-%m-%dT%H:%MZ in files read using read_csv().

### To Reproduce

For example:

```
import duckdb
import pandas as pd

url = ['https://storage.googleapis.com/neon-publication/NEON.DOM.SITE.DP4.00132.001/TOOK/20210801T000000--20210901T000000/expanded/NEON.D18.TOOK.DP4.00132.001.bat_sonarRecord.2021-08.expanded.20251206T025505Z.csv']
schema = {'uid': 'VARCHAR',
'domainID': 'VARCHAR',
'siteID': 'VARCHAR',
'namedLocation': 'VARCHAR',
'startDate': 'TIMESTAMPTZ',
'endDate': 'TIMESTAMPTZ',
'eventID': 'VARCHAR',
'sonarRecordingStartTime': 'TIMESTAMPTZ',
'sonarRecordingStopTime': 'TIMESTAMPTZ',
'sonarRecordingNumber': 'VARCHAR',
'remarks': 'VARCHAR',
'dataQF': 'VARCHAR'}
tformat = "'%Y-%m-%dT%H:%MZ'"
dat = duckdb.sql(f"SELECT * FROM read_csv({url}, header=true, columns={schema}, timestampformat={tformat})")
d = dat.df()
d[0:3]
```

uid | domainID | siteID | namedLocation | startDate | endDate | eventID | sonarRecordingStartTime | sonarRecordingStopTime | sonarRecordingNumber | remarks | dataQF
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
b72ef2b3-45f0-492f-85a2-03f7c65c1cb6 | D18 | TOOK | TOOK | NaT | NaT | TOOK.2021 | NaT | NaT | TOOK_20210806_131828.dt4 | None | None
6d46d5cf-124e-430e-b1ba-fb5ebb1a01f0 | D18 | TOOK | TOOK | NaT | NaT | TOOK.2021 | NaT | NaT | TOOK_20210806_103707.dt4 | None | None
a61d20e3-b179-4c18-9eb0-124afc04e1d5 | D18 | TOOK | TOOK | NaT | NaT | TOOK.2021 | NaT | NaT | TOOK_20210804_164251.dt4 | None | None

The four date-time fields are all empty (NaT) in the output here.

I am not seeing this problem on dates without times, or on timestamps that include seconds. This example includes seconds and correctly parses the time stamps:

```
url = ['https://storage.googleapis.com/neon-publication/NEON.DOM.SITE.DP1.00005.001/MOAB/20250501T000000--20250601T000000/basic/NEON.D13.MOAB.DP1.00005.001.000.010.030.IRBT_30_minute.2025-05.basic.20250715T232027Z.csv']
schema = {'startDateTime': 'TIMESTAMPTZ',
'endDateTime': 'TIMESTAMPTZ',
'bioTempMean': 'DOUBLE',
'bioTempMinimum': 'DOUBLE',
'bioTempMaximum': 'DOUBLE',
'bioTempVariance': 'DOUBLE',
'bioTempNumPts': 'DOUBLE',
'bioTempExpUncert': 'DOUBLE',
'bioTempStdErMean': 'DOUBLE',
'finalQF': 'BIGINT'}
tformat = "'%Y-%m-%dT%H:%M:%SZ'"
dat = duckdb.sql(f"SELECT * FROM read_csv({url}, header=true, columns={schema}, timestampformat={tformat})")
d = dat.df()
d[0:3]
```

startDateTime | endDateTime | bioTempMean | bioTempMinimum | bioTempMaximum | bioTempVariance | bioTempNumPts | bioTempExpUncert | bioTempStdErMean | finalQF
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
2025-04-30 18:00:00-06:00 | 2025-04-30 18:30:00-06:00 | 20.78 | 19.35 | 22.14 | 0.80 | 1800.0 | 0.54 | 0.02 | 0
2025-04-30 18:30:00-06:00 | 2025-04-30 19:00:00-06:00 | 15.47 | 10.56 | 19.35 | 10.81 | 1800.0 | 0.57 | 0.08 | 0
2025-04-30 19:00:00-06:00 | 2025-04-30 19:30:00-06:00 | 12.71 | 11.77 | 13.53 | 0.30 | 1800.0 | 0.54 | 0.01 | 0

I have also been able to correctly parse time stamps in format %Y-%m-%dT%H:%MZ using the R implementation of duckdb, so I think this is specific to the Python package.

### OS:

macOS 26.4.1

### DuckDB Package Version:

1.5.2

### Python Version:

3.13

### Full Name:

Claire Lunch

### Affiliation:

National Ecological Observatory Network

### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

### Did you include all relevant data sets for reproducing the issue?

Yes

### Did you include all code required to reproduce the issue?

- [x] Yes, I have

### Did you include all relevant configuration to reproduce the issue?

- [x] Yes, I have

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.