instacart / instacart/lore

save_weights fails when large number of input features are present

Open
#95 4 comments 0 reactions 0 assignees View on GitHub
bug enhancement
Dominant language
Python
Stars
1.5k
Forks
128
PR merge metrics
No merged PRs in 30d

Description

Hi @montanalow . This is really a great work. I really like how you abstract the common pitfalls in machine learning and streamline the process in this project. I see a lot of potential in this project from a data scientist perspective. If you don't mind, I can provide my feedback from using this tool.

For this particular issue, I encountered `h5py` error because of too many `Input` layers. As show [here](https://github.com/instacart/lore/blob/master/lore/estimators/keras.py#L189-L201), we have to pass one encoder for each column in the dataframe, and each encoder corresponds to one `Input` layer. I deal with a lot of DNA sequence data which is usually >5000 columns. I think it makes sense to at least combine the columns using `Continuous` or `Pass` encoders into one `Input`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.