dotnet / dotnet/machinelearning
Make DataFrame ReadCsv more efficient
Open
area-DataFrame
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
DataFrame.ReadCsv at the moment loops over all the rows in a file twice: once to find the number of rows and schema, and the 2nd time to append the data. Each line is read in as a string and `string.Split` is called on it. Profiling shows that a sizeable chunk of time is spend in the `Split` calls. We should explore using a ReadOnlySpan to parse the file and avoid all the intermediate strings.
Contributor guide
Assessment
This issue has not been assessed yet.