aws / aws/amazon-sagemaker-examples

[Bug Report] AutoPilot Transformer Model - Unexpected number of columns

Open
#2,855 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I have created my own AutoPilot experiment following the tutorial [here](https://aws.amazon.com/blogs/machine-learning/creating-high-quality-machine-learning-models-for-financial-services-using-amazon-sagemaker-autopilot/). I was able to create the transformer model yet when I call:
```
transformer.transform(data=test_data_path,
split_type='Line',
content_type='text/csv',
wait=True)
```
I get the following error:
`Received data of unknown size. Expected number of columns is 654. Number of columns in the received data is 653.`

This error, however, makes no sense to me because 654 is the number of columns in the train dataset, and this transformer is supposed to work on the test dataset which does not contain the label (and thus 653 is the expected number of columns):
```
>> train_df.shape
(312715, 654)
>> test_df.shape
(337737, 653)
```
Unfortunately, I cannot share the data publicly but any advice on how to address this error would be appreciated.

Contributor guide

Open the contributing guide

Research direction

Start with the transformer.transform call shown in the report and compare the train and test dataset schemas, including the missing label column. The issue is resolved when the required input schema for the AutoPilot transformer is established and the 653-column test data is either accepted or the necessary preprocessing is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, machine-learning
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.