Azure / Azure/MachineLearningNotebooks

Pandas dataframes with array column values are not correctly persisted as AzureML datasets

Open
#1,587 0 comments 0 reactions 0 assignees View on GitHub
ADO bug MLOps
Dominant language
Jupyter Notebook
Stars
4.4k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

Pandas dataframes with arrays as column values seem to be incorrectly persisted. An example:

```python
test_df = pd.DataFrame({'x': [np.random.rand(1000) for _ in range(1000)]})
ds = Datastore.get_default(ws)
Dataset.Tabular.register_pandas_dataframe(test_df, ds, 'test_dataset')

test_df.head()
###
x
0 [0.5044850335733219, 0.6054305053424696, 0.669...
1 [0.41759815476145723, 0.266477750018155, 0.511...
2 [0.6777708610872593, 0.16925324567267985, 0.16...
3 [0.4268294269387616, 0.6540643485117185, 0.033...
4 [0.6560106490417036, 0.5804652379458484, 0.582...

Dataset.get_by_name(ws, 'test_dataset').to_pandas_dataframe().head()
###
x
0 ERROR
1 ERROR
2 ERROR
3 ERROR
4 ERROR
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.