dotnet / dotnet/machinelearning
Question: improve performance of reading files?
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
I am running LightGbmBinaryTrainer through AutoML API. The start of training is slow.
Are there ways to make it faster? CPU usage is less than 1 core, HDD reading is also very low.
The subsequent experiments seem much faster.
For example, should I read the data to memory before starting AutoML.
```
var loadOptions = columnInference.TextLoaderOptions;
loadOptions.UseThreads = true;
TextLoader textLoader = mlContext.Data.CreateTextLoader(loadOptions);
```
This is where it seems to spend most of time, on every pause. I have not run profiling though (it has some issues)

Contributor guide
Assessment
This issue has not been assessed yet.