alteryx / alteryx/evalml

Spike: Preserve woodwork schema through dask

Aperta
#2,252 2 commenti 1 reazione 1 assegnatario Rivendicata da @asniyaz Vedi su GitHub
refactor spike
Lingua principale
Python
Stelle
850
Fork
96
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

In #2243, we added `X_schema` and `y_schema` arguments to `train_pipeline` and `score_pipeline` because the dask engine would lose the schema once it pickles the dataframe and sends it to the workers.

The underlying problem is that `pickle` does not preserve the woodwork schema. This sounds like a general accessor thing and is probably not specific to woodwork.

```python
import pandas as pd
import woodwork as ww
import pickle
import pytest

df = pd.DataFrame({"a": [1, 2, 3], 'id': [0, 1, 2]}, index=[4, 5, 6])
df.ww.init()
assert df.ww.schema is not None

assert pickle.loads(pickle.dumps(df)).ww.schema is None
```

This issue tracks figuring out how to make dask serialize woodwork data structures. Once we come up with a solution, we can discuss if this should live in evalml or woodwork.

Some resources to check out:

- [Extending dask serialization](https://distributed.dask.org/en/latest/serialization.html#id3)
- [woodwork serializers](https://woodwork.alteryx.com/en/stable/generated/woodwork.serialize.typing_info_to_dict.html#woodwork.serialize.typing_info_to_dict)

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.