aws / aws/amazon-sagemaker-examples

Import error in advanced_functionality/autogluon-tabular/AutoGluon_Tabular_SageMaker.ipynb

Open
#1,208 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Description of the error**

when running the following cell in [AutoGluon_Tabular_SageMaker.ipynb](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/advanced_functionality/autogluon-tabular):
```python
%%time

instance_type = 'ml.m5.2xlarge'
#instance_type = 'local'

ecr_image = f'{ecr_uri_prefix}/{training_algorithm_name}:latest'

estimator = Estimator(image_name=ecr_image,
role=role,
train_instance_count=1,
train_instance_type=instance_type,
hyperparameters=hyperparameters)

estimator.fit(train_s3_path)
```

I got the following error:
```
UnexpectedStatusException: Error for Training job autogluon-sagemaker-training-2020-05-10-07-12-25-701: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command "/usr/local/bin/python3.6 train.py --label y"
```

The log provided by training container was bellow.
```
Traceback (most recent call last):
File "train.py", line 20, in
import autogluon as ag
File "package/autogluon/__init__.py", line 10, in
from . import scheduler, searcher, utils
File "package/autogluon/scheduler/__init__.py", line 6, in
from .fifo import *
File "package/autogluon/scheduler/fifo.py", line 17, in
from ..searcher import BaseSearcher
File "package/autogluon/searcher/__init__.py", line 2, in
from .skopt_searcher import *
File "package/autogluon/searcher/skopt_searcher.py", line 6, in
from skopt import Optimizer
File "package/skopt/__init__.py", line 55, in
from .searchcv import BayesSearchCV
File "package/skopt/searchcv.py", line 16, in
from sklearn.utils.fixes import MaskedArray
ImportError: cannot import name 'MaskedArray'
2020-05-11 01:47:03,374 sagemaker-containers ERROR ExecuteUserScriptError:
Command "/usr/local/bin/python3.6 train.py --label y"
Traceback (most recent call last):
File "/usr/local/bin/dockerd-entrypoint.py", line 20, in
subprocess.check_call(shlex.split(' '.join(sys.argv[1:])))
File "/usr/local/lib/python3.6/subprocess.py", line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['train']' returned non-zero exit status 1.
```

**To reproduce**
1. Create the SageMaker notebook instance
- Create new role which can access any S3 bucket
- Set Git repository option to clone the amazon-sagemaker example repository to the notebook instance only.
2. Run [the notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/autogluon-tabular/AutoGluon_Tabular_SageMaker.ipynb)

Contributor guide

Open the contributing guide

Research direction

Open advanced_functionality/autogluon-tabular/AutoGluon_Tabular_SageMaker.ipynb and start with the training cell that creates the Estimator and calls estimator.fit(train_s3_path). Reproduce the import failure and inspect the logged sklearn.utils.fixes MaskedArray error. Done means the notebook's training step runs without this import error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.