【Feature Discussion】Make tb_plugin support larger traces

Hi guys, I'm recently working on profiling LLM (like GPT-3) workloads using the Pytorch profiler. When I tried to visualize the trace using tensorboard, one major problem was that the trace is too large (typically > 1GB for one step), causing no response in the browser.

I would like to work on this problem to make the tb_plugin support larger traces. Below are a few of my concerns:

• I cannot find a doc for tb_plugin (about the code structure and the plugin's arch), which makes it difficult for me to get started.
• Could you provide some insight into this problem? like where is the possible bottleneck (the original tensorboard? the trace analysis server? the frontend browser? or something else?)
• I also noticed that "monitoring daemon for larger scale deployments" is in progress, is this solving the same problem? If yes, can I get involved?

Thanks : )

The issue names no files or tests; start by locating the tb_plugin entry points and comparing them with the monitoring daemon mentioned in the discussion. Done means establishing how traces larger than 1 GB are handled and making the visualization responsive, with coverage for the relevant large-trace path.