[Python] Dataset Timezone Handling

I'm trying to write a pandas dataframe with a datetimeindex with timezone information to a pyarrow dataset but the timezone information doesn't seem to be written (apart from in the pandas metadata)

 

For example

 
```java

import os
import pandas as pd
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds

from pathlib import Path

# I've tried with both v2.0 and v3.0 today
print(pa.__version__)

# create dummy dataframe with datetime index containing tz info
df = pd.DataFrame(
dict(
timestamp=pd.date_range("2021-01-01", freq="1T", periods=100, tz="US/Eastern"),
x=np.arange(100),
)
).set_index("timestamp")

test_dir = Path("test_dir")
table = pa.Table.from_pandas(df)
schema = table.schema

print(schema)
print(schema.pandas_metadata)

# warning - creates dir in cwd
pq.write_to_dataset(table, test_dir)

# timestamp column is us and UTC
print(pq.ParquetFile(test_dir / os.listdir(test_dir)[0]).read())

# create dataset using schema from earlier
dataset = ds.dataset(test_dir, format="parquet", schema=schema)

# doesn't work
dataset.to_table()
```
 

 

Is this a bug or am I missing something?

Thanks

Andy

 

**Reporter**: [Andy Douglas](https://issues.apache.org/jira/browse/ARROW-11388)

_{**Note**: *This issue was originally created as [ARROW-11388](https://issues.apache.org/jira/browse/ARROW-11388). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*}

Start by running the supplied pandas and PyArrow reproduction, comparing pq.ParquetFile(...).read() with ds.dataset(...).to_table() when the pandas-derived schema is passed. Trace the dataset and Parquet handling reached by pq.write_to_dataset and ds.dataset. Done means the timezone information remains available after reading the dataset, rather than only in pandas metadata.

Written by the indexing model from the issue text.