aws / aws/amazon-sagemaker-examples
[Bug Report] AutoPilot Transformer Model - Unexpected number of columns
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
I have created my own AutoPilot experiment following the tutorial [here](https://aws.amazon.com/blogs/machine-learning/creating-high-quality-machine-learning-models-for-financial-services-using-amazon-sagemaker-autopilot/). I was able to create the transformer model yet when I call:
```
transformer.transform(data=test_data_path,
split_type='Line',
content_type='text/csv',
wait=True)
```
I get the following error:
`Received data of unknown size. Expected number of columns is 654. Number of columns in the received data is 653.`
This error, however, makes no sense to me because 654 is the number of columns in the train dataset, and this transformer is supposed to work on the test dataset which does not contain the label (and thus 653 is the expected number of columns):
```
>> train_df.shape
(312715, 654)
>> test_df.shape
(337737, 653)
```
Unfortunately, I cannot share the data publicly but any advice on how to address this error would be appreciated.
Contributor guide
Research direction
Start with the transformer.transform call shown in the report and compare the train and test dataset schemas, including the missing label column. The issue is resolved when the required input schema for the AutoPilot transformer is established and the 653-column test data is either accepted or the necessary preprocessing is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, machine-learning
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100