Graph size balloons over consecutive Keras models

Running many unrelated Keras models in one Python script leads to
steadily increasing event file sizes, even when the models and callbacks
are unrelated to each other.

I noticed this because I ran 200 runs overnight, and wondered why the
event file for the first one was 220KB while the event file for the last
run was 20MB.

Run the following script in tf-nightly-2.0-preview in Python 3 (in a
directory where you don’t mind the logs directory being erased):

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import os
import shutil

import numpy as np
import tensorflow as tf


LOGDIR = "logs"  # will be erased

INPUT_SHAPE = (10,)
OUTPUT_CLASSES = 10


def model_fn():
  model = tf.keras.models.Sequential([
      tf.keras.layers.Input(INPUT_SHAPE),
      tf.keras.layers.Dense(64, activation="relu"),
      tf.keras.layers.Dense(OUTPUT_CLASSES, activation="softmax"),
  ])
  model.compile(optimizer="adam", loss="sparse_categorical_crossentropy")
  return model


def make_data(m):
  x = np.random.uniform(size=(m,) + INPUT_SHAPE)
  y = np.random.randint(0, OUTPUT_CLASSES, size=(m,))
  return (x, y)


def main():
  np.random.seed(0)
  (x_train, y_train) = make_data(1000)
  (x_test, y_test) = make_data(100)
  shutil.rmtree(LOGDIR, ignore_errors=True)
  for i in range(10):
    model = model_fn()
    callback = tf.keras.callbacks.TensorBoard(
        os.path.join(LOGDIR, str(i)),
        update_freq=600,
        profile_batch=0,
    )
    model.fit(x=x_train, y=y_train, callbacks=[callback])


if __name__ == "__main__":
  main()

Then run wc -c logs/**/train/*:

$ wc -c logs/**/train/*
  53584 logs/0/train/events.out.tfevents.1554484297.HOSTNAME.131202.562.v2
 106858 logs/1/train/events.out.tfevents.1554484297.HOSTNAME.131202.1350.v2
 160140 logs/2/train/events.out.tfevents.1554484298.HOSTNAME.131202.2131.v2
 213422 logs/3/train/events.out.tfevents.1554484298.HOSTNAME.131202.2912.v2
 266704 logs/4/train/events.out.tfevents.1554484298.HOSTNAME.131202.3693.v2
 320258 logs/5/train/events.out.tfevents.1554484299.HOSTNAME.131202.4474.v2
 373810 logs/6/train/events.out.tfevents.1554484299.HOSTNAME.131202.5255.v2
 427362 logs/7/train/events.out.tfevents.1554484300.HOSTNAME.131202.6036.v2
 480914 logs/8/train/events.out.tfevents.1554484300.HOSTNAME.131202.6817.v2
 534469 logs/9/train/events.out.tfevents.1554484300.HOSTNAME.131202.7598.v2
2937521 total

In each case, the graph makes up the vast majority of the event file
(all but a kilobyte or so):

RUN     SIZE OF GRAPH
---     -------------
0         52367 bytes
1        105637 bytes
2        158919 bytes
3        212201 bytes
4        265483 bytes
5        319035 bytes
6        372587 bytes
7        426139 bytes
8        479691 bytes
9        533246 bytes