6+ Hours Server Startup Time

Hello, I am currently experiencing a very annoying issue with tensorboard.
Basically, in order to open any events file/folder (usually done with tensorboard --logdir XXX, I have to wait an insanely long time (a folder with 12 tensorbard files takes more than six hours at 100% CPU load in order to open). My machine is pretty beefy, so that's not the issue here. There is definetely something weird going on either with my logging files or my tensorboard installation.

First of all I uninstalled/reinstalled tensorboard on a completely new environment, by running pip install -U tensorboard. I tried many different TB versions and the issue is still there. I am currently using version 2.11.

Secondly I tried to replicate the issue on a much smaller file with just some random scalars and some images. The file weighs only some MB, definetely something that should be handled very smoothly. Loading times for this file is approximately 90s (still way too long). It also appears to scale non-linearly with the events file size, meaning that twice the file size results in more than twice the loading time.

I then wrote a simple script that loads an events file from python and measured the loading time:

from tensorboard.backend.event_processing.event_accumulator import EventAccumulator

f = "/a/very/personal/path/to/a/tensorboard/logging/folder"
event_acc = EventAccumulator(f)
event_acc.Reload()

And then I went deeper by profiling the program execution with cProfile.
Apparently, 97% of the time is wasted in pywrap_tensorboard.py:crc_update, invoked 78610 times with an average 1.1ms per-call time.

I attach the cProfile outputs, so you can take a look.
The file haz ZIP extension but it is not really a zip file, this is just to make it pass github bs file checks. You can open it with any python profiler viewers like snakeviz.

pip install snakeviz
snakeviz PATH/TO/FILE

god_why.zip

Start with tensorboard.backend.event_processing.event_accumulator.EventAccumulator and the pywrap_tensorboard.py:crc_update path described in the cProfile output. Reproduce the slow Reload() on the supplied event data and profile it; done means the excessive loading time and CRC-related cost are understood and addressed, with loading performance verified.