awslabs / awslabs/sagemaker-debugger
tf.keras saves step at end of batch
- Dominant language
- Python
- Stars
- 165
- Forks
- 82
- PR merge metrics
- No merged PRs in 30d
Description
Running the following script with tensorflow==1.15.0:
```
import tensorflow.compat.v2 as tf
import smdebug.tensorflow as smd
from tempfile import TemporaryDirectory
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255, x_test / 255
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax'),
])
with TemporaryDirectory() as dirpath:
hook = smd.KerasHook(out_dir=dirpath)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5, callbacks=[hook])
trial = smd.create_trial(path=dirpath)
print(hook)
print(trial)
```
gives the following output:
```
:(
out_dir=/var/folders/r1/mgxfss8d45jbs_vl464bbsg906jznv/T/tmpdzybvlqg,
tensorboard_dir=None,
step=9374,
mode=ModeKeys.TRAIN,
mode_steps={: 9374, : 9374},
include_collections=['metrics', 'losses', 'sm_metrics'],
writer=None,
save_config=: , : ,
reduction_config=,
save_all=False,
dry_run=False,
)
:(
name=tmpdzybvlqg,
path=/var/folders/r1/mgxfss8d45jbs_vl464bbsg906jznv/T/tmpdzybvlqg,
steps=[0, 500, 1000, 1500, 1874, 2000, 2500, 3000, 3500, 3749, 4000, 4500, 5000, 5500, 5624, 6000, 6500, 7000, 7499, 7500, 8000, 8500, 9000, 9374],
collections=['default', 'weights', 'biases', 'gradients', 'losses', 'metrics', 'inputs', 'outputs', 'all', 'sm_metrics'],
tensor_names=['acc', 'batch', 'loss', 'size'],
)
```
It appears to be saving every 1874th step, in addition to every 500th. Is this desired behavior?
Contributor guide
Research direction
Start with smd.KerasHook and its callback integration with model.fit, then inspect how the hook records steps and how smd.create_trial reports them. Reproduce the TensorFlow 1.15 script and determine whether the additional end-of-batch checkpoints are intended; done means the behavior is confirmed and the issue is clarified with the relevant outcome.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100