aws / aws/amazon-sagemaker-examples

[Bug Report] Sentiment Analysis with Apache MXNet and Gluon: ValueError: max() arg is an empty sequence

Open
#3,142 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-python-sdk/mxnet_gluon_sentiment/mxnet_sentiment_analysis_with_gluon.ipynb via https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-python-sdk/mxnet_gluon_sentiment/mxnet_sentiment_analysis_with_gluon.html

**Describe the bug**
Attempting to run the MXNet Gluon Sentiment example, except modified to NOT use the example data or a Jupyter notebook, Python script provides its own labeled training and test data and otherwise uses all other settings from the script. I also use the `sentiment.py` script [from this repo](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-python-sdk/mxnet_gluon_sentiment/sentiment.py), as the training data script is not clear at all.

An error around max() is encounter, it starts ~line 200 in the `sentiment.py` script. The training and test data I have is very small, around 300 or so labeled examples. Could that be causing the issue?

**To reproduce**
Relevant part of script that is giving the issue

```python
def train_mxnet():
'''
This function uses the SageMaker API to create a training Job for Apache MXNet classifier
'''
# Call "prepare_training_data" to retrieve the un-labeled half of the tweets
prepare_training_data()

# wget https://raw.githubusercontent.com/aws/amazon-sagemaker-examples/master/sagemaker-python-sdk/mxnet_gluon_sentiment/sentiment.py

# Upload test data to Sagemaker bucket
bucket = sagemaker.session.Session().default_bucket()
inputs = sagemaker.s3.S3Uploader.upload(
'./data',
f's3://{bucket}/GluonCVESentiment/data'
)

print(f'Sagemaker S3 Inputs of data complete')

m = MXNet(
'sentiment.py',
role=sagemaker.get_execution_role(),
instance_count=1,
instance_type='ml.c4.xlarge',
framework_version='1.8.0',
py_version='py37',
distribution={
'parameter_server': {
'enabled': True
}
},
hyperparameters={
'batch-size': 8,
'epochs': 2,
'learning-rate': 0.01,
'embedding-size': 50,
'log-interval': 1000,
}
)
print(m)
```

**Logs**

From SageMaker.

```
2022-02-07 20:14:18,659 sagemaker-training-toolkit INFO Imported framework sagemaker_mxnet_container.training
--
2022-02-07 20:14:18,661 sagemaker-training-toolkit INFO No GPUs detected (normal if no gpus installed)
2022-02-07 20:14:18,683 sagemaker_mxnet_container.training INFO MXNet training environment: {'SM_HOSTS': '["algo-1"]', 'SM_NETWORK_INTERFACE_NAME': 'eth0', 'SM_HPS': '{"batch-size":8,"embedding-size":50,"epochs":2,"learning-rate":0.01,"log-interval":1000}', 'SM_USER_ENTRY_POINT': 'sentiment.py', 'SM_FRAMEWORK_PARAMS': '{"sagemaker_parameter_server_enabled":true}', 'SM_RESOURCE_CONFIG': '{"current_group_name":"homogeneousCluster","current_host":"algo-1","current_instance_type":"ml.c4.xlarge","hosts":["algo-1"],"instance_groups":[{"hosts":["algo-1"],"instance_group_name":"homogeneousCluster","instance_type":"ml.c4.xlarge"}],"network_interface_name":"eth0"}', 'SM_INPUT_DATA_CONFIG': '{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}}', 'SM_OUTPUT_DATA_DIR': '/opt/ml/output/data', 'SM_CHANNELS': '["training"]', 'SM_CURRENT_HOST': 'algo-1', 'SM_MODULE_NAME': 'sentiment', 'SM_LOG_LEVEL': '20', 'SM_FRAMEWORK_MODULE': 'sagemaker_mxnet_container.training:main', 'SM_INPUT_DIR': '/opt/ml/input', 'SM_INPUT_CONFIG_DIR': '/opt/ml/input/config', 'SM_OUTPUT_DIR': '/opt/ml/output', 'SM_NUM_CPUS': '4', 'SM_NUM_GPUS': '0', 'SM_MODEL_DIR': '/opt/ml/model', 'SM_MODULE_DIR': 's3://sagemaker-us-east-2-ACCOUNTID/mxnet-training-2022-02-07-20-11-38-533/source/sourcedir.tar.gz', 'SM_TRAINING_ENV': '{"additional_framework_parameters":{"sagemaker_parameter_server_enabled":true},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_mxnet_container.training:main","hosts":["algo-1"],"hyperparameters":{"batch-size":8,"embedding-size":50,"epochs":2,"learning-rate":0.01,"log-interval":1000},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"mxnet-training-2022-02-07-20-11-38-533","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-us-east-2-ACCOUNTID/mxnet-training-2022-02-07-20-11-38-533/source/sourcedir.tar.gz","module_name":"sentiment","network_interface_name":"eth0","num_cpus":4,"num_gpus":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_group_name":"homogeneousCluster","current_host":"algo-1","current_instance_type":"ml.c4.xlarge","hosts":["algo-1"],"instance_groups":[{"hosts":["algo-1"],"instance_group_name":"homogeneousCluster","instance_type":"ml.c4.xlarge"}],"network_interface_name":"eth0"},"user_entry_point":"sentiment.py"}', 'SM_USER_ARGS': '["--batch-size","8","--embedding-size","50","--epochs","2","--learning-rate","0.01","--log-interval","1000"]', 'SM_OUTPUT_INTERMEDIATE_DIR': '/opt/ml/output/intermediate', 'SM_CHANNEL_TRAINING': '/opt/ml/input/data/training', 'SM_HP_BATCH-SIZE': '8', 'SM_HP_EMBEDDING-SIZE': '50', 'SM_HP_EPOCHS': '2', 'SM_HP_LEARNING-RATE': '0.01', 'SM_HP_LOG-INTERVAL': '1000'}
2022-02-07 20:14:18,705 sagemaker_mxnet_container.training INFO Starting distributed training task
2022-02-07 20:14:19,397 sagemaker-training-toolkit INFO No GPUs detected (normal if no gpus installed)
2022-02-07 20:14:19,412 sagemaker-training-toolkit INFO No GPUs detected (normal if no gpus installed)
2022-02-07 20:14:19,426 sagemaker-training-toolkit INFO No GPUs detected (normal if no gpus installed)
2022-02-07 20:14:19,440 sagemaker-training-toolkit INFO Invoking user script
Training Env:
{ "additional_framework_parameters": { "sagemaker_parameter_server_enabled": true }, "channel_input_dirs": { "training": "/opt/ml/input/data/training" }, "current_host": "algo-1", "framework_module": "sagemaker_mxnet_container.training:main", "hosts": [ "algo-1" ], "hyperparameters": { "batch-size": 8, "embedding-size": 50, "epochs": 2, "learning-rate": 0.01, "log-interval": 1000 }, "input_config_dir": "/opt/ml/input/config", "input_data_config": { "training": { "TrainingInputMode": "File", "S3DistributionType": "FullyReplicated", "RecordWrapperType": "None" } }, "input_dir": "/opt/ml/input", "is_master": true, "job_name": "mxnet-training-2022-02-07-20-11-38-533", "log_level": 20, "master_hostname": "algo-1", "model_dir": "/opt/ml/model", "module_dir": "s3://sagemaker-us-east-2-ACCOUNTID/mxnet-training-2022-02-07-20-11-38-533/source/sourcedir.tar.gz", "module_name": "sentiment", "network_interface_name": "eth0", "num_cpus": 4, "num_gpus": 0, "output_data_dir": "/opt/ml/output/data", "output_dir": "/opt/ml/output", "output_intermediate_dir": "/opt/ml/output/intermediate", "resource_config": { "current_host": "algo-1", "current_instance_type": "ml.c4.xlarge", "current_group_name": "homogeneousCluster", "hosts": [ "algo-1" ], "instance_groups": [ { "instance_group_name": "homogeneousCluster", "instance_type": "ml.c4.xlarge", "hosts": [ "algo-1" ] } ], "network_interface_name": "eth0" }, "user_entry_point": "sentiment.py"
}
Environment variables:
SM_HOSTS=["algo-1"]
SM_NETWORK_INTERFACE_NAME=eth0
SM_HPS={ "batch-size": 8, "embedding-size": 50, "epochs": 2, "learning-rate": 0.01, "log-interval": 1000 }
SM_USER_ENTRY_POINT=sentiment.py
SM_FRAMEWORK_PARAMS={ "sagemaker_parameter_server_enabled": true }
SM_RESOURCE_CONFIG={ "current_group_name": "homogeneousCluster", "current_host": "algo-1", "current_instance_type": "ml.c4.xlarge", "hosts": [ "algo-1" ], "instance_groups": [ { "hosts": [ "algo-1" ], "instance_group_name": "homogeneousCluster", "instance_type": "ml.c4.xlarge" } ], "network_interface_name": "eth0" }
SM_INPUT_DATA_CONFIG={ "training": { "RecordWrapperType": "None", "S3DistributionType": "FullyReplicated", "TrainingInputMode": "File" } }
SM_OUTPUT_DATA_DIR=/opt/ml/output/data
SM_CHANNELS=["training"]
SM_CURRENT_HOST=algo-1
SM_MODULE_NAME=sentiment
SM_LOG_LEVEL=20
SM_FRAMEWORK_MODULE=sagemaker_mxnet_container.training:main
SM_INPUT_DIR=/opt/ml/input
SM_INPUT_CONFIG_DIR=/opt/ml/input/config
SM_OUTPUT_DIR=/opt/ml/output
SM_NUM_CPUS=4
SM_NUM_GPUS=0
SM_MODEL_DIR=/opt/ml/model
SM_MODULE_DIR=s3://sagemaker-us-east-2-ACCOUNTID/mxnet-training-2022-02-07-20-11-38-533/source/sourcedir.tar.gz
SM_TRAINING_ENV={ "additional_framework_parameters": { "sagemaker_parameter_server_enabled": true }, "channel_input_dirs": { "training": "/opt/ml/input/data/training" }, "current_host": "algo-1", "framework_module": "sagemaker_mxnet_container.training:main", "hosts": [ "algo-1" ], "hyperparameters": { "batch-size": 8, "embedding-size": 50, "epochs": 2, "learning-rate": 0.01, "log-interval": 1000 }, "input_config_dir": "/opt/ml/input/config", "input_data_config": { "training": { "RecordWrapperType": "None", "S3DistributionType": "FullyReplicated", "TrainingInputMode": "File" } }, "input_dir": "/opt/ml/input", "is_master": true, "job_name": "mxnet-training-2022-02-07-20-11-38-533", "log_level": 20, "master_hostname": "algo-1", "model_dir": "/opt/ml/model", "module_dir": "s3://sagemaker-us-east-2-ACCOUNTID/mxnet-training-2022-02-07-20-11-38-533/source/sourcedir.tar.gz", "module_name": "sentiment", "network_interface_name": "eth0", "num_cpus": 4, "num_gpus": 0, "output_data_dir": "/opt/ml/output/data", "output_dir": "/opt/ml/output", "output_intermediate_dir": "/opt/ml/output/intermediate", "resource_config": { "current_group_name": "homogeneousCluster", "current_host": "algo-1", "current_instance_type": "ml.c4.xlarge", "hosts": [ "algo-1" ], "instance_groups": [ { "hosts": [ "algo-1" ], "instance_group_name": "homogeneousCluster", "instance_type": "ml.c4.xlarge" } ], "network_interface_name": "eth0" }, "user_entry_point": "sentiment.py" }
SM_USER_ARGS=["--batch-size","8","--embedding-size","50","--epochs","2","--learning-rate","0.01","--log-interval","1000"]
SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
SM_CHANNEL_TRAINING=/opt/ml/input/data/training
SM_HP_BATCH-SIZE=8
SM_HP_EMBEDDING-SIZE=50
SM_HP_EPOCHS=2
SM_HP_LEARNING-RATE=0.01
SM_HP_LOG-INTERVAL=1000
PYTHONPATH=/opt/ml/code:/usr/local/bin:/usr/local/lib/python37.zip:/usr/local/lib/python3.7:/usr/local/lib/python3.7/lib-dynload:/usr/local/lib/python3.7/site-packages
Invoking script with the following command:
/usr/local/bin/python3.7 sentiment.py --batch-size 8 --embedding-size 50 --epochs 2 --learning-rate 0.01 --log-interval 1000
WARNING: discarded 17 sentences longer than the largest bucket.
WARNING: discarded 0 sentences longer than the largest bucket.
Traceback (most recent call last): File "sentiment.py", line 385, in args.embedding_size, File "sentiment.py", line 74, in train val_iterator = BucketSentenceIter(val_sentences, val_labels, batch_size) File "sentiment.py", line 200, in __init__ self.default_bucket_key = max(buckets)
ValueError: max() arg is an empty sequence
2022-02-07 20:14:23,198 sagemaker-training-toolkit ERROR Reporting training FAILURE
2022-02-07 20:14:23,199 sagemaker-training-toolkit ERROR ExecuteUserScriptError:
ExitCode 1
ErrorMessage ""
Command "/usr/local/bin/python3.7 sentiment.py --batch-size 8 --embedding-size 50 --epochs 2 --learning-rate 0.01 --log-interval 1000"
2022-02-07 20:14:23,199 sagemaker-training-toolkit ERROR Encountered exit_code 1
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.