dotnet / dotnet/machinelearning

Improve error experience when LoadCsv throws

Open
#5,656 3 comments 3 reactions 0 assignees View on GitHub
area-DataFrame
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

If you have .csv file that has some error in the data, for example:

```
Date | Daily minimum temperatures
1/1/1981 | 20.7
1/2/1981 | 17.9
1/3/1981 | 18.8
1/4/1981 | 14.6
1/5/1981 | 15.8
...
7/20/1982 | ?0.2
7/21/1982 | ?0.8
```

Here you get an exception:

```
System.FormatException: Input string was not in a correct format.
at System.Number.ThrowOverflowOrFormatException(ParsingStatus status, TypeCode type)
at System.String.System.IConvertible.ToSingle(IFormatProvider provider)
at System.Convert.ChangeType(Object value, Type conversionType, IFormatProvider provider)
at Microsoft.Data.Analysis.DataFrame.Append(IEnumerable`1 row, Boolean inPlace)
at Microsoft.Data.Analysis.DataFrame.LoadCsv(Stream csvStream, Char separator, Boolean header, String[] columnNames, Type[] dataTypes, Int64 numberOfRowsToRead, Int32 guessRows, Boolean addIndexColumn, Encoding encoding)
```

Which isn't super helpful to tell you where or what was causing the problem. We should log a more helpful error in this situation so users know what is wrong.

We should also consider having a mode where things like this turn into `N/A` values, like `Single.Nan`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.