aws / aws/amazon-sagemaker-examples
Training fails on huggingface_byo_scripts_and_data
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
[huggingface_byo_scripts_and_data](https://github.com/aws/amazon-sagemaker-examples/blob/main/advanced_functionality/huggingface_byo_scripts_and_data/huggingface-custom-text-summarizer.ipynb)
**Describe the bug**
Running the notebook training fails with message:
``
MPI_ABORT was invoked on rank 2 in communicator MPI COMMUNICATOR 5 DUP FROM 0
with errorcode 1.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
``
**To reproduce**
Run the notebook above.
**Logs**
Error from the sdk that run the job:
```
UnexpectedStatusException Traceback (most recent call last)
Cell In[19], line 1
----> 1 huggingface_estimator.fit({"train": f"s3://{session_bucket}/{s3_prefix}/train/"})
File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/workflow/pipeline_context.py:272, in runnable_by_pipeline..wrapper(*args, **kwargs)
268 return context
270 return _StepArguments(retrieve_caller_name(self_instance), run_func, *args, **kwargs)
--> 272 return run_func(*args, **kwargs)
File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/estimator.py:1156, in EstimatorBase.fit(self, inputs, wait, logs, job_name, experiment_config)
1154 self.jobs.append(self.latest_training_job)
1155 if wait:
-> 1156 self.latest_training_job.wait(logs=logs)
File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/estimator.py:2297, in _TrainingJob.wait(self, logs)
2295 # If logs are requested, call logs_for_jobs.
2296 if logs != "None":
-> 2297 self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
2298 else:
2299 self.sagemaker_session.wait_for_job(self.job_name)
File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/session.py:4214, in Session.logs_for_job(self, job_name, wait, poll, log_type)
4211 last_profiler_rule_statuses = profiler_rule_statuses
4213 if wait:
-> 4214 self._check_job_status(job_name, description, "TrainingJobStatus")
4215 if dot:
4216 print()
File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/session.py:3747, in Session._check_job_status(self, job, desc, status_key_name)
3741 if "CapacityError" in str(reason):
3742 raise exceptions.CapacityError(
3743 message=message,
3744 allowed_statuses=["Completed", "Stopped"],
3745 actual_status=status,
3746 )
-> 3747 raise exceptions.UnexpectedStatusException(
3748 message=message,
3749 allowed_statuses=["Completed", "Stopped"],
3750 actual_status=status,
3751 )
UnexpectedStatusException: Error for Training job huggingface-summarizer-2023-03-08-06-03-07-199: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command "mpirun --host algo-1 -np 8 --allow-run-as-root --tag-output --oversubscribe -mca btl_tcp_if_include eth0 -mca oob_tcp_if_include eth0 -mca plm_rsh_no_tree_spawn 1 -mca pml ob1 -mca btl ^openib -mca orte_abort_on_non_zero_status 1 -mca btl_vader_single_copy_mechanism none -mca plm_rsh_num_concurrent 1 -x NCCL_SOCKET_IFNAME=eth0 -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH -x SMDATAPARALLEL_USE_SINGLENODE=1 -x FI_PROVIDER=efa -x RDMAV_FORK_SAFE=1 -x LD_PRELOAD=/opt/conda/lib/python3.6/site-packages/gethostname.cpython-36m-x86_64-linux-gnu.so smddprun /opt/conda/bin/python3.6 -m mpi4py train.py --epoch 1 --model-name google/pegasus-xsum --target-column title --text-column review_text"
b7905.enderr>:[nltk_data] Downloading package punkt to /root/nltk_data...
b7915.enderr>:[nltk_data] Downloading package punkt to /root/nltk_data...
b7945.enderr>:[nltk_data] Downloading package punkt to /root/nltk_data...
b7975.enderr>:[nltk_data] Downloading package punkt to /root/
```
Contributor guide
Research direction
Start with advanced_functionality/huggingface_byo_scripts_and_data/huggingface-custom-text-summarizer.ipynb and reproduce the failure at the huggingface_estimator.fit call. Inspect the training job logs and the notebook's distributed training setup; done means the notebook's training job completes without the MPI_ABORT or ExecuteUserScriptError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, huggingface, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100