aws-samples / aws-samples/sagemaker-101-workshop

MNIST model CPU training broken in TF v2.7 (conda_tensorflow2_p37 kernel on NBI ALv2 JLv3)

Open
#28 0 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Jupyter Notebook
Stars
80
Forks
51
PR merge metrics
No merged PRs in 30d

Description

The current `conda_tensorflow2_p38` kernel on the latest SageMaker Notebook Instance platform (`notebook-al2-v2`, as used in the CFn template) seems to break local CPU-only training for the MNIST migration challenge.

In this environment (TF v2.7.1, TF.Keras v2.7.0), `tensorflow.keras.backend.image_data_format()` asks for `channels_first`, but training fails because MaxPoolingOp only supports channels_last on CPU - per the error message below:

```
InvalidArgumentError: Default MaxPoolingOp only supports NHWC on device type CPU
[[node sequential/max_pooling2d/MaxPool
(defined at /home/ec2-user/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/keras/layers/pooling.py:357)
]] [Op:__inference_train_function_862]

Errors may have originated from an input operation.
Input Source operations connected to node sequential/max_pooling2d/MaxPool:
In[0] sequential/conv2d_1/Relu (defined at /home/ec2-user/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/keras/backend.py:4867)
```

Overriding the `image_data_format()` check (in "Pre-Process the Data for our CNN") to prepare data in different shape does not work because the model is incompatible (will raise ValueError in conv2d_2).

Still seems to be working fine in current SMStudio kernel (TensorFlow v2.3.2, TF.Keras v2.4.0).

Contributor guide

Open the contributing guide

Research direction

Start with the MNIST migration challenge's “Pre-Process the Data for our CNN” section and compare its behavior in the TensorFlow 2.3.2 and 2.7.1 kernels. Investigate the reported CPU MaxPoolingOp and Conv2D shape errors; done means the challenge trains locally on CPU in the current SageMaker kernel without an incompatible data-shape override.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.