tensorflow / tensorflow/tensorboard

Huge memory consumption to inspect a TensorBoard Profile.

Open
#6,132 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

Environment information (required)

Diagnostics output 
--- check: autoidentify
INFO: diagnose_tensorboard.py version 516a2f9433ba4f9c3a4fdb0f89735870eda054a1

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=10, micro=6, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='martijn-desktop', release='5.15.0-57-generic', version='#63~20.04.1-Ubuntu SMP Wed Nov 30 13:40:16 UTC 2022', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tensorboard==2.11.0
INFO: installed: tensorflow==2.11.0
INFO: installed: tensorflow-estimator==2.11.0
INFO: installed: tensorboard-data-server==0.6.1

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.11.0'

--- check: tensorflow_python_version
2023-01-05 18:59:35.166853: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-05 18:59:35.943019: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:
2023-01-05 18:59:35.943095: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:
2023-01-05 18:59:35.943106: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
INFO: tensorflow.__version__: '2.11.0'
INFO: tensorflow.__git_version__: 'v2.11.0-rc2-17-gd5b57ca93e5'

--- check: tensorboard_data_server_version
INFO: data server binary: '/home/martijn/.local/lib/python3.10/site-packages/tensorboard_data_server/bin/server'
INFO: data server binary version: b'rustboard 0.6.1'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/home/martijn/.local/bin/tensorboard\n'

--- check: addrinfos
socket.has_ipv6 = True
socket.AF_UNSPEC = <AddressFamily.AF_UNSPEC: 0>
socket.SOCK_STREAM = <SocketKind.SOCK_STREAM: 1>
socket.AI_ADDRCONFIG = <AddressInfo.AI_ADDRCONFIG: 32>
socket.AI_PASSIVE = <AddressInfo.AI_PASSIVE: 1>
Loopback flags: <AddressInfo.AI_ADDRCONFIG: 32>
Loopback infos: [(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::1', 0, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0))]
Wildcard flags: <AddressInfo.AI_PASSIVE: 1>
Wildcard infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('0.0.0.0', 0)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::', 0, 0, 0))]

--- check: readable_fqdn
INFO: socket.getfqdn(): 'martijn-desktop'

--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=4464738, st_dev=66306, st_nlink=2, st_uid=1000, st_gid=1000, st_size=4096, st_atime=1672924927, st_mtime=1672940892, st_ctime=1672940892)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/home/martijn/.local/lib/python3.10/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==1.3.0
appdirs==1.4.4
apturl==0.5.2
arandr==0.1.10
astunparse==1.6.3
attrs==21.2.0
autobahn==21.11.1
Automat==20.2.0
base58==1.0.3
bcrypt==3.2.0
beautifulsoup4==4.10.0
beniget==0.4.1
blessed==1.19.1
blinker==1.4
bottle==0.12.19
Brlapi==0.8.3
Brotli==1.0.9
cachetools==5.2.0
cbor==1.0.0
certifi==2020.6.20
cffi==1.15.0
chardet==4.0.0
click==8.0.3
colorama==0.4.4
command-not-found==0.3
constantly==15.1.0
cryptography==3.4.8
cupshelpers==1.0
cycler==0.11.0
Cython==0.29.28
dbus-python==1.2.18
decorator==4.4.2
defer==1.0.6
defusedxml==0.7.1
distro==1.7.0
distro-info===1.1build1
docker==5.0.3
docker-pycreds==0.4.0
duplicity==0.8.21
ecdsa==0.18.0b1
fasteners==0.14.1
flatbuffers==22.12.6
fonttools==4.29.1
fs==2.4.12
future==0.18.2
gast==0.4.0
GeoIP==1.3.2
gitdb==4.0.10
GitPython==3.1.30
Glances==3.2.4.2
google-auth==2.15.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
gpustat==1.0.0
greenlet==2.0.1
grpcio==1.51.1
gviz-api==1.10.0
h5py==3.7.0
hkdf==0.0.3
html5lib==1.1
httplib2==0.20.2
humanize==0.0.0
hyperlink==21.0.0
idna==3.3
importlib-metadata==4.6.4
incremental==21.3.0
influxdb==5.3.1
jeepney==0.7.1
kazam==1.4.5
keras==2.11.0
keyring==23.5.0
kiwisolver==1.3.2
language-selector==0.1
launchpadlib==1.10.16
lazr.restfulclient==0.14.4
lazr.uri==1.0.6
libclang==14.0.6
lockfile==0.12.2
louis==3.20.0
lxml==4.8.0
lz4==3.1.3+dfsg
macaroonbakery==1.3.1
magic-wormhole==0.12.0
Mako==1.1.3
Markdown==3.4.1
MarkupSafe==2.1.1
matplotlib==3.5.1
meson==0.61.2
mnemonic==0.19
monotonic==1.6
more-itertools==8.10.0
mpmath==0.0.0
msgpack==1.0.3
neovim==0.3.1
netifaces==0.11.0
numpy==1.21.5
nvidia-ml-py==11.495.46
oauthlib==3.2.0
olefile==0.46
opt-einsum==3.3.0
packaging==21.3
paramiko==2.9.3
passlib==1.7.4
pathtools==0.1.2
pexpect==4.8.0
Pillow==9.0.1
pip==22.0.2
ply==3.11
promise==2.3
protobuf==3.20.0
psutil==5.9.0
ptyprocess==0.7.0
py-ubjson==0.16.1
pyasn1==0.4.8
pyasn1-modules==0.2.1
pycairo==1.20.1
pycparser==2.21
pycryptodomex==3.11.0
pycups==2.0.1
Pygments==2.11.2
PyGObject==3.42.1
PyHamcrest==2.0.2
PyJWT==2.3.0
pymacaroons==0.13.0
PyNaCl==1.5.0
pynvim==0.4.3
pyOpenSSL==21.0.0
pyparsing==2.4.7
pypng==0.0.20
PyQRCode==1.2.1
pyRFC3339==1.1
pysmi==0.3.2
pysnmp==4.4.12
pystache==0.6.0
python-apt==2.3.0+ubuntu2.1
python-dateutil==2.8.1
python-debian===0.1.43ubuntu1
python-snappy==0.5.3
pythran==0.10.0
PyTrie==0.4.0
pytz==2022.1
pyxattr==0.7.2
pyxdg==0.27
PyYAML==5.4.1
reportlab==3.6.8
requests==2.25.1
requests-oauthlib==1.3.1
rsa==4.9
scipy==1.8.0
scour==0.38.2
screen-resolution-extra==0.0.0
SecretStorage==3.3.1
sentry-sdk==1.12.1
service-identity==18.1.0
setproctitle==1.3.2
setuptools==59.6.0
shortuuid==1.0.11
six==1.16.0
smmap==5.0.0
sortedcontainers==2.1.0
soupsieve==2.3.1
spake2==0.8
ssh-import-id==5.11
sympy==1.9
systemd-python==234
tensorboard==2.11.0
tensorboard-data-server==0.6.1
tensorboard-plugin-profile==2.11.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.11.0
tensorflow-estimator==2.11.0
tensorflow-io-gcs-filesystem==0.29.0
termcolor==2.1.1
termplotlib==0.3.9
tqdm==4.57.0
Twisted==22.1.0
txaio==21.2.1
txtorcon==20.0.0
typing_extensions==4.4.0
u-msgpack-python==2.3.0
ubuntu-advantage-tools==27.12
ubuntu-drivers-common==0.0.0
ufoLib2==0.13.1
ufw==0.36.1
ujson==5.1.0
unattended-upgrades==0.1
unicodedata2==14.0.0
urllib3==1.26.13
usb-creator==0.3.7
wadllib==1.3.6
wandb==0.13.7
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.2.3
Werkzeug==2.2.2
wheel==0.37.1
wrapt==1.14.1
wsaccel==0.6.3
xdg==5
xkit==0.0.0
youtube-dl==2021.12.17
zipp==1.0.0
zope.interface==5.4.0
  • Google Chrome: Version 108.0.5359.124 (Official Build) (64-bit)

Issue description

Profiling a TF process using the Tracing tools in TF produces this huge .xplane.pb file (around 1GB for my case). Opening this in TensorBoard works, but literally consumes 20GB to 30GB of RAM. I'm mostly interested in opening the tracing information (with Google Chrome). In the past we had access to a .trace.json.gz file, which contained all we need. Now it seems wrapped in this Protobuf file, which TensorBoard has to load. Loading this seems to consume 30GB for me, and makes me actually wait around 40 seconds for this thing to appear. In the past it was instant. Now the ironic part is that once the data I'm after (the tracing information) is loaded into the trace viewer from Chrome, it only consumes 1GB (which is an amount I would consider almost reasonable).

image

In the end, I don't know why stuff changed in TF2 with the profiling data. But this seems a step backwards. Also this memory footprint is unacceptable. The tool to load some timeline events takes up 20GB.

I'd like to be able to just extract the .trace.json.gz as a workaround, while hoping for an actual fix for this issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue in TensorBoard's Profile tracing tools with the reported 1GB .xplane.pb input, and compare its memory use and load time with the extracted .trace.json.gz in Chrome. Done means the tracing information loads without the reported 20–30GB memory consumption, or the trace data can be extracted directly as a documented workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
tensorflow
Domain
data-visualization, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.