aws-samples / aws-samples/sagemaker-run-notebook

Processing Job not showing failure of notebook fails

Open
#53 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
144
Forks
72
PR merge metrics
No merged PRs in 30d

Description

Hi,
We discovered that even when a notebook causes an error, the processing job stops in status "complete".
What we would need is in the case of a notebook having an exception to show a failed state.
I had a look at the execute.py and found this section:
https://github.com/aws-samples/sagemaker-run-notebook/blob/master/sagemaker_run_notebook/container/execute.py#L92
```python
except Exception as e:
# Write out an error file. This will be returned as the failureReason in the
# DescribeProcessingJob result.
trc = traceback.format_exc()
# with open(os.path.join(output_path, 'failure'), 'w') as s:
# s.write('Exception during processing: ' + str(e) + '\n' + trc)
# Printing this causes the exception to be in the training job logs, as well.
print("Exception during processing: " + str(e) + "\n" + trc, file=sys.stderr)
# A non-zero exit code causes the training job to be marked as Failed.
# sys.exit(255)
output_notebook = "xyzzy" # Dummy for print, below
```

I tried to change it to the following and rebuilding the docker image, but the processing job still shows "complete" instead of "failed" on a failure.
```python
except Exception as e:
# Write out an error file. This will be returned as the failureReason in the
# DescribeTrainingJob result.
trc = traceback.format_exc()
with open(os.path.join(os.path.dirname(output_notebook), 'failure'), 'w') as s:
s.write('Exception during execution: ' + str(e) + '\n' + trc)
# Printing this causes the exception to be in the training job logs, as well.
print('Exception during execution: ' + str(e) + '\n' + trc, file=sys.stderr)
# A non-zero exit code causes the training job to be marked as Failed.
sys.exit(255)
```

Contributor guide

Open the contributing guide

Research direction

Start in sagemaker_run_notebook/container/execute.py at the exception handler and trace how the exit status and failure file affect the SageMaker processing job. Reproduce the issue with a notebook that raises an exception; done when the processing job reports Failed instead of Complete and exposes the failure details.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.