dotnet / dotnet/machinelearning

GetTempPath hot in some benchmarks in Microsoft.ML.PerformanceTests

Open
#7,277 1 comment 0 reactions 0 assignees View on GitHub
area-Core
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

I have noticed that depending on the length of the temporary path, some benchmarks see a significant performance drop because of the need to convert from a Windows short name to a long name.

If I use the temporary directory path `C:\tempdir\`, the benchmark gives these performance results on my machine:

```
Microsoft.ML.PerformanceTests.exe --filter '*CreatePredictionEngine' --envVars TEMP:C:\tempdir TMP:C:\tempdir

| Method | Mean | Error | StdDev | Extra Metric |
|----------------------- |---------:|--------:|--------:|-------------:|
| CreatePredictionEngine | 249.5 us | 4.80 us | 4.93 us | - |
```

If I use a long temporary directory `C:\Users\username\AppData\Local\Temp\ML.NET\A Very Very Long File Name That Will Be Converted To A Short Path\Temp` (which has the short name `C:\Users\username\AppData\Local\Temp\ML.NET\AVERYV~1\Temp`), the benchmark shows significantly worse performance on the same machine:

```
.\Microsoft.ML.PerformanceTests.exe --filter '*CreatePredictionEngine' --envVars TEMP:C:\Users\username\AppData\Local\Temp\ML.NET\AVERYV~1\Temp TMP:C:\Users\username\AppData\Local\Temp\ML.NET\AVERYV~1\Temp

| Method | Mean | Error | StdDev | Extra Metric |
|----------------------- |---------:|---------:|---------:|-------------:|
| CreatePredictionEngine | 10.53 ms | 0.082 ms | 0.077 ms | - |
```

Looking at this under PerfView, I see a lot of extra time spent [here](https://github.com/dotnet/machinelearning/blob/f385b06aa0aeb64000fc341ecc31e43057c1ee13/src/Microsoft.ML.Core/Environment/HostEnvironmentBase.cs#L330).

I'm wondering if it is possible to cache this lookup? It seems misleading for the benchmark performance to fluctuate so much because of the temporary directory path.

Contributor guide

Open the contributing guide

Research direction

Start with the lookup at src/Microsoft.ML.Core/Environment/HostEnvironmentBase.cs around line 330, then reproduce the issue using Microsoft.ML.PerformanceTests.exe with the CreatePredictionEngine filter and the two TEMP/TMP paths shown. Compare the benchmark results and determine whether caching the temporary-path lookup removes the performance fluctuation; done means the benchmark no longer incurs repeated path-conversion overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
performance, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.