Optimize non-dense TensorSpan operations to avoid temporary buffer allocation
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- PR merge metrics
- PR metrics pending
Description
## Tracking: Avoid temporary buffer allocation and optimize access patterns for non-dense TensorSpan
### Summary
Non-dense TensorSpan operations (e.g., `SequenceEqual`, `FillGaussianNormalDistribution`, `FillUniformDistribution`, `Resize`, `IndexOfMax`, `ConcatenateOnDimension`) currently rely on temporary buffer allocation as a workaround for handling logical iteration. While this ensures correctness, it can add significant performance overhead due to memory access/copying, especially in hot paths or with large tensors.
**Tanner Gooding's feedback in PR #124225:**
> Getting it working correctly is most important, so I think the temp buffer for right now is ok, but we should have a tracking issue to avoid the temporary buffer since it can add significant expense (due to amount of memory accesses/copying)
>
> In general this is meant to be handled via the TensorDimensionSpan iterator, which allows getting the contiguous sequences more easily -or- via the TensorShape.AdjustToNextIndex APIs for more complex cases.
### Additional suggestions from PR #125582
In https://github.com/dotnet/runtime/pull/125582#pullrequestreview-3961984905 :
- Review of `TensorOperation.Invoke` code paths and their interaction with buffer allocation.
- Consider whether APIs can expose contiguous chunks appropriately for performance, and whether element-wise iteration can fall back to span when possible.
### Action Items
- Investigate and implement allocation-free approaches for non-dense tensor operations using `TensorDimensionSpan` or related APIs.
- Ensure performance parity between dense and non-dense fast/slow paths.
- Review usage across all affected public APIs, including those listed above, for correct and efficient logical element iteration.
### References
- [PR #124225 Feedback](https://github.com/dotnet/runtime/pull/124225#discussion_r2947784721)
- [PR #125582 Review](https://github.com/dotnet/runtime/pull/125582#pullrequestreview-3961984905)
Contributor guide
Assessment
This issue has not been assessed yet.