dotnet / dotnet/machinelearning

GC pause caused by Tuple<int, int> on model load

Open
#6,807 1 comment 0 reactions 0 assignees View on GitHub
area-TorchSharp enhancement untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**Is your feature request related to a problem? Please describe.**
My app is quite sensitive to GC pauses, so I've been doing allocation profiling. I noticed that a lot of GC pauses are being caused by the usage of `Tuple` within the `Attention` class when the model is loaded.

https://github.com/dotnet/machinelearning/blob/aaf226c7e7c359edf27e663362e928e02c8b9d0f/src/Microsoft.ML.TorchSharp/AutoFormerV2/Attention.cs#L72-L79

**Describe the solution you'd like**
In my own fork, I've switched to using `ValueTuple`, which has removed the allocations and thus GC pauses

**Describe alternatives you've considered**
I don't think there's an easier/cleaner fix for this.

**Additional context**
Using `Microsoft.ML.TorchSharp` version `0.21.0-preview.23266.6`, I observed 240k allocations of `Tuple` during model loading, coming in just after `Tensor`.

ml-allocations

After switching to `ValueTuple` on my own fork, these allocations were removed, resulting in a significant reduction in CG pauses during model loading.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.