aws-samples / aws-samples/sagemaker-101-workshop
MNIST model CPU training broken in TF v2.7 (conda_tensorflow2_p37 kernel on NBI ALv2 JLv3)
- Dominant language
- Jupyter Notebook
- Stars
- 80
- Forks
- 51
- PR merge metrics
- No merged PRs in 30d
Description
The current `conda_tensorflow2_p38` kernel on the latest SageMaker Notebook Instance platform (`notebook-al2-v2`, as used in the CFn template) seems to break local CPU-only training for the MNIST migration challenge.
In this environment (TF v2.7.1, TF.Keras v2.7.0), `tensorflow.keras.backend.image_data_format()` asks for `channels_first`, but training fails because MaxPoolingOp only supports channels_last on CPU - per the error message below:
```
InvalidArgumentError: Default MaxPoolingOp only supports NHWC on device type CPU
[[node sequential/max_pooling2d/MaxPool
(defined at /home/ec2-user/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/keras/layers/pooling.py:357)
]] [Op:__inference_train_function_862]
Errors may have originated from an input operation.
Input Source operations connected to node sequential/max_pooling2d/MaxPool:
In[0] sequential/conv2d_1/Relu (defined at /home/ec2-user/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/keras/backend.py:4867)
```
Overriding the `image_data_format()` check (in "Pre-Process the Data for our CNN") to prepare data in different shape does not work because the model is incompatible (will raise ValueError in conv2d_2).
Still seems to be working fine in current SMStudio kernel (TensorFlow v2.3.2, TF.Keras v2.4.0).
Contributor guide
Research direction
Start with the MNIST migration challenge's “Pre-Process the Data for our CNN” section and compare its behavior in the TensorFlow 2.3.2 and 2.7.1 kernels. Investigate the reported CPU MaxPoolingOp and Conv2D shape errors; done means the challenge trains locally on CPU in the current SageMaker kernel without an incompatible data-shape override.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100