dotnet / dotnet/machinelearning

Support Multiple Threads on ML Related DB Access

Open
#6,692 1 comment 0 reactions 0 assignees View on GitHub
area-Core enhancement needs-further-triage untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**Is your feature request related to a problem? Please describe.**
According to my tests, when trying to Execute multiple `RegressionExperiments` in parallel where data is loaded through `DatabaseSource`, some bottleneck occurs on the DB access layer, probably due to a single instance of the DB access layer (even when creating a separate `DatabaseSource` for each experiment).

This prevents accessing the full throttle of the cpu and the database.

My tests were:
1. Reviewing DB logs: when running experiments, the DB access occurs sequentially and not at the same time.
2. Running from separate EXE: when running each experiment from a separate process, suddenly the DB gets much more queries at the same time. all experiments end faster.

Still I might miss something so correct me if I am wrong.

**Describe the solution you'd like**
Allow instantiation of multiple DB connections to allow running experiments in parallel.

**Describe alternatives you've considered**

- Tried to load the idataview with the entire db data and then dropcolumns or filter but that took very long time (loading from db)

**Additional context**
Windows machine, both database and ml are on the same machine.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.