dotnet / dotnet/machinelearning

Add line limit to readMultilines in TextLoader

Open
#5,144 0 comments 0 reactions 0 assignees View on GitHub
area-Core enhancement Priority:2 usability
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

_(This issue tracks @justinormont 's suggestion [here](https://github.com/dotnet/machinelearning/pull/5125#discussion_r427002585))_

Recent PR #5125 added a `readMultilines `option to `TextLoader `to enable the posibility of including newlines inside quoted fields.

A problem with this is that if the input file isn't correctly formatted (i.e., if it has a quote that opens a quoted field, that is never closed) then the `Multilinereader `will actually load every line until it finds another quote. Depending on the dataset (and on how many incorrectly formatted rows it has) it could actually load into memory the whole dataset (or as much as the `StringBuilder `supports, which is [typically 2^32 chars ](https://docs.microsoft.com/en-us/dotnet/api/system.text.stringbuilder.maxcapacity?view=netcore-3.1#System_Text_StringBuilder_MaxCapacity))

For example:
```
id,description,animal
0,"this quoted field isnt closed,cat
1,this field doesnt include quotes,dog
... // no quoted fields in here
2555,"it is until this quoted field that the multilinereader actually stops reading row 0",bird
2556,"this row will be read correctly",dog
```

@justinormont 's suggestion here: https://github.com/dotnet/machinelearning/pull/5125#discussion_r427002585

is to add another option to the `TextLoader `that let the user set the maximum length of a row, and if that threshold is passed, then simply ignore the line and continue reading the input file without loading everything into it.

I think that before introducing more options to the `TextLoader`, it's better to see if users actually hit this problem when using `readMultilines`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.