aws-samples / aws-samples/sagemaker-run-notebook
Processing Job not showing failure of notebook fails
- Dominant language
- Python
- Stars
- 144
- Forks
- 72
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
We discovered that even when a notebook causes an error, the processing job stops in status "complete".
What we would need is in the case of a notebook having an exception to show a failed state.
I had a look at the execute.py and found this section:
https://github.com/aws-samples/sagemaker-run-notebook/blob/master/sagemaker_run_notebook/container/execute.py#L92
```python
except Exception as e:
# Write out an error file. This will be returned as the failureReason in the
# DescribeProcessingJob result.
trc = traceback.format_exc()
# with open(os.path.join(output_path, 'failure'), 'w') as s:
# s.write('Exception during processing: ' + str(e) + '\n' + trc)
# Printing this causes the exception to be in the training job logs, as well.
print("Exception during processing: " + str(e) + "\n" + trc, file=sys.stderr)
# A non-zero exit code causes the training job to be marked as Failed.
# sys.exit(255)
output_notebook = "xyzzy" # Dummy for print, below
```
I tried to change it to the following and rebuilding the docker image, but the processing job still shows "complete" instead of "failed" on a failure.
```python
except Exception as e:
# Write out an error file. This will be returned as the failureReason in the
# DescribeTrainingJob result.
trc = traceback.format_exc()
with open(os.path.join(os.path.dirname(output_notebook), 'failure'), 'w') as s:
s.write('Exception during execution: ' + str(e) + '\n' + trc)
# Printing this causes the exception to be in the training job logs, as well.
print('Exception during execution: ' + str(e) + '\n' + trc, file=sys.stderr)
# A non-zero exit code causes the training job to be marked as Failed.
sys.exit(255)
```
Contributor guide
Research direction
Start in sagemaker_run_notebook/container/execute.py at the exception handler and trace how the exit status and failure file affect the SageMaker processing job. Reproduce the issue with a notebook that raises an exception; done when the processing job reports Failed instead of Complete and exposes the failure details.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100