dotnet / dotnet/machinelearning

Question: improve performance of reading files?

Open
#6,024 1 comment 0 reactions 0 assignees View on GitHub
area-Core
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

I am running LightGbmBinaryTrainer through AutoML API. The start of training is slow.
Are there ways to make it faster? CPU usage is less than 1 core, HDD reading is also very low.
The subsequent experiments seem much faster.

For example, should I read the data to memory before starting AutoML.

```
var loadOptions = columnInference.TextLoaderOptions;
loadOptions.UseThreads = true;
TextLoader textLoader = mlContext.Data.CreateTextLoader(loadOptions);
```

This is where it seems to spend most of time, on every pause. I have not run profiling though (it has some issues)
![image](https://user-images.githubusercontent.com/26261427/146651154-22686c8c-fb47-4cb1-8a0f-712a84e78134.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.