[tf 2.0] use summary apis without occupying gpu memory

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

System information

• TensorFlow version (you are using): 2.0.0-dev20190725

Describe the feature and the current behavior/state.
With current tf 2.0 summary in a gpu-equipped machine, the process will occupy gpu memory as follows:

In [1]: import tensorflow as tf

In [2]: tf.__version__
Out[2]: '2.0.0-dev20190725'

In [3]: !nvidia-smi
Tue Jul 30 21:47:26 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.48                 Driver Version: 410.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  Off  | 00000000:3B:00.0 Off |                    0 |
| N/A   28C    P0    26W / 250W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-PCIE...  Off  | 00000000:AF:00.0 Off |                    0 |
| N/A   28C    P0    25W / 250W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

In [4]: writer = tf.summary.create_file_writer("./testdir")
...
info be omitted to avoid clutter
...

In [5]: with writer.as_default():
   ...:     tf.summary.scalar("test", 123, step=1)
   ...:

In [6]: !nvidia-smi
Tue Jul 30 21:47:44 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.48                 Driver Version: 410.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  Off  | 00000000:3B:00.0 Off |                    0 |
| N/A   29C    P0    41W / 250W |  15460MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-PCIE...  Off  | 00000000:AF:00.0 Off |                    0 |
| N/A   29C    P0    36W / 250W |    418MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     63963      C   ...2.0/envs/tf-nightly-2.0-0725/bin/python 15449MiB |
|    1     63963      C   ...2.0/envs/tf-nightly-2.0-0725/bin/python   407MiB |
+-------------------------------------

We expect some apis which we can use to log metrics without gpu usage, such as tf.Summary and tf.Summary.Value in tf 1.x.

Will this change the current api? How?
Not sure.

Who will benefit with this feature?
Who wants to use tensorflow apis to log results in a separate process, which is expected to occupy no GPU memory.

The same issue was opened and then closed in https://github.com/tensorflow/tensorflow/issues/31165