awslabs / awslabs/sagemaker-debugger

tf.keras saves step at end of batch

Open
#144 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
165
Forks
82
PR merge metrics
No merged PRs in 30d

Description

Running the following script with tensorflow==1.15.0:

```
import tensorflow.compat.v2 as tf
import smdebug.tensorflow as smd
from tempfile import TemporaryDirectory

mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255, x_test / 255

model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax'),
])

with TemporaryDirectory() as dirpath:
hook = smd.KerasHook(out_dir=dirpath)

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5, callbacks=[hook])

trial = smd.create_trial(path=dirpath)
print(hook)
print(trial)
```

gives the following output:

```
:(
out_dir=/var/folders/r1/mgxfss8d45jbs_vl464bbsg906jznv/T/tmpdzybvlqg,
tensorboard_dir=None,
step=9374,
mode=ModeKeys.TRAIN,
mode_steps={: 9374, : 9374},
include_collections=['metrics', 'losses', 'sm_metrics'],
writer=None,
save_config=: , : ,
reduction_config=,
save_all=False,
dry_run=False,
)
:(
name=tmpdzybvlqg,
path=/var/folders/r1/mgxfss8d45jbs_vl464bbsg906jznv/T/tmpdzybvlqg,
steps=[0, 500, 1000, 1500, 1874, 2000, 2500, 3000, 3500, 3749, 4000, 4500, 5000, 5500, 5624, 6000, 6500, 7000, 7499, 7500, 8000, 8500, 9000, 9374],
collections=['default', 'weights', 'biases', 'gradients', 'losses', 'metrics', 'inputs', 'outputs', 'all', 'sm_metrics'],
tensor_names=['acc', 'batch', 'loss', 'size'],
)
```

It appears to be saving every 1874th step, in addition to every 500th. Is this desired behavior?

Contributor guide

Open the contributing guide

Research direction

Start with smd.KerasHook and its callback integration with model.fit, then inspect how the hook records steps and how smd.create_trial reports them. Reproduce the TensorFlow 1.15 script and determine whether the additional end-of-batch checkpoints are intended; done means the behavior is confirmed and the issue is clarified with the relevant outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.