aws / aws/amazon-sagemaker-examples

mxnet-realtime-analysis.ipynb failed CI

Open
#2,353 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

Link to the notebook:
https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-debugger/mxnet_realtime_analysis/mxnet-realtime-analysis.ipynb

Error:

---------------------------------------------------------------------------
Exception encountered at "In [9]":
---------------------------------------------------------------------------
TensorUnavailableForStep Traceback (most recent call last)
in
31 steps_to_render = list(set(steps).symmetric_difference(set(rendered_steps)))
32 # plot only tensors from newer chunk
---> 33 plot_tensors(trial, layer, batch_index, steps_to_render)
34 rendered_steps.extend(steps_to_render)
35 time.sleep(5)

in plot_tensors(trial, layer, batch_index, steps_range)
18 if len(steps_range) > 0:
19 fig, axs = create_plots(steps_range)
---> 20 vals = get_data(trial, layer, batch_index, steps_range)
21
22 for ax, image, step in

[...]

TensorUnavailableForStep: Value for tensor conv0_output_0 is not available for step 1000 with mode GLOBAL as it was not saved.

Contributor guide

Open the contributing guide

Research direction

Open sagemaker-debugger/mxnet_realtime_analysis/mxnet-realtime-analysis.ipynb and inspect In [9], especially plot_tensors and get_data. Run the notebook through the repository's CI process to reproduce the TensorUnavailableForStep error. Done means the notebook completes without requesting tensor data that was not saved.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook
Domain
machine-learning, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.