dotnet / dotnet/machinelearning

KMeansTrainer.Options NumberOfThreads has no effect? Low CPU load during .Fit()

Open
#7,477 1 comment 0 reactions 0 assignees View on GitHub
area-Trainers untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**System Information:**
- Windows 11 22H2
- ML.NET Version: Microsoft.ML 4.0.2
- .NET Version: .NET 9.0

**Describe the bug**
I have a large dataset with approx. 500,000 spectras with 200 data points each. I use the kMeansTrainer for the clustering. It works - but during the "training" (which takes ~75 s) ...

```csharp
var model = pipeline.Fit(trainingData);
```
... the CPU load is less than 20% on a i7-11850H @ 2.5 GHz CPU with 16 threads. If I got the ML.Net code right, it allows max. 16 / 2 = 8 threads. But I can set the `NumberOfThreads` in the trainer options from 1 to 8 - it doesn't change anything.

```csharp
var options = new KMeansTrainer.Options
{
NumberOfClusters = numberOfClusters,
// InitializationAlgorithm = KMeansTrainer.InitializationAlgorithm.KMeansYinyang,
// OptimizationTolerance = 1e-7f, // default 1e-7f
NumberOfThreads = 8, // default 8
// MaximumNumberOfIterations = 1000
};
var pipeline = mlContext.Clustering.Trainers.KMeans(options);
```
The consumed time doesn't change and the CPU load is also low.

If I reduce e.g. the `MaximumNumberOfIterations` or the `OptimizationTolerance` I can see an effect on the training time (and on the results).

For the normalization of the data I use Parallel.For loops and during this loops, I can clearly see that all thread ramp up to ~80% load for a short time.

If I dig into the code: [KMeansLloydsYinYangTrain.Train](https://github.com/dotnet/machinelearning/blob/aea5651f13e18c4387266e830224abe25f7e9bf1/src/Microsoft.ML.KMeansClustering/KMeansPlusPlusTrainer.cs#L1351) in every iteration the [chunks should be processes in parallel](https://github.com/dotnet/machinelearning/blob/aea5651f13e18c4387266e830224abe25f7e9bf1/src/Microsoft.ML.KMeansClustering/KMeansPlusPlusTrainer.cs#L1394), right? If I select the maximum of the 8 threads, I would expect that 8 threads should run at max. load.

**Expected behavior**
I expect that the KMeans training should use the maximum allowed CPU power in order to reduce time.

**Additional context**
We compared our C# app with MatLab kMeans with the same data: ML.Net is faster than the MatLab implementation, but if I see that the cpu has 80 % idle time during the training, it could be even faster. ;)

Thanks,
Mike

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.