Azure wrongly reads Parquet

Setup

Python=3.8 + azureml-core=1.36.0 + azureml-dataprep=2.26.0 + pyarrow=7.0.0 + pandas=1.4

Summary

to_pandas_dataframe wrongly reads certain Parquet datasets. Data of some columns appears to be internally shuffled.
This was already reported but closed without a fix, due to issues with sharing data publicly.
I share the reproducible example below

How to reproduce

from azureml.core import Workspace, Dataset
import tempfile
import pandas as pd

# prepare data: list of sha-values with some None values
df = pd.read_csv('error_data.csv')

# configure Azure storage
ws = Workspace.from_config()
dstore = ws.datastores.get('your datastore')
dstore_path = 'relative datastore path'
target = (dstore,dstore_path)

# write to Azure storage
with tempfile.TemporaryDirectory() as tmpdir:
    df.to_parquet(f'{tmpdir}/df.parquet')
    ds=Dataset.File.upload_directory(tmpdir,target,overwrite=True)

# read by two ways: download and open in pandas or use the Azure connector
with tempfile.TemporaryDirectory() as tmpdir:
    ds=Dataset.File.from_files(target)
    ds.download(tmpdir)
    df1 = pd.read_parquet(tmpdir)
    ds = Dataset.Tabular.from_parquet_files(target)
    df2 = ds.to_pandas_dataframe()

# comparison fails, the data seems displaced :-(
pd.testing.assert_frame_equal(df1,df2)

error_data.csv

從 issue 中的可重現範例開始，將下載的 pandas DataFrame 與 Dataset.Tabular.from_parquet_files(...).to_pandas_dataframe() 進行比較。追蹤 Azure Dataset 的表格式 Parquet 讀取路徑，然後使用提供的 error_data.csv，透過 pd.testing.assert_frame_equal(df1, df2) 驗證完成情況。