dotnet / dotnet/machinelearning

Make DataFrame ReadCsv more efficient

Open
#5,685 3 comments 0 reactions 1 assignee Claimed by @pgovind View on GitHub
area-DataFrame
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

DataFrame.ReadCsv at the moment loops over all the rows in a file twice: once to find the number of rows and schema, and the 2nd time to append the data. Each line is read in as a string and `string.Split` is called on it. Profiling shows that a sizeable chunk of time is spend in the `Split` calls. We should explore using a ReadOnlySpan to parse the file and avoid all the intermediate strings.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.