dotnet / dotnet/machinelearning

Precision loss When Reading in CSV data

Open
#6,347 3 comments 0 reactions 0 assignees View on GitHub
area-DataFrame enhancement needs-further-triage
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

### Summary
CSV fields not auto- reading In values correctly (losing precision)

### Scenario
I want to automatically read some Integer Id's into my dataframe. These integers are quite large and are losing precision.

### Cause
https://github.com/dotnet/machinelearning/blob/main/src/Microsoft.Data.Analysis/DataFrame.IO.cs
***Method: GuessType()***
We don't make any attempt to read Int, Long or Double Types here. Hence the code-flow is truncating the long integers.
I guess since this is primarily a ML framework, we favour floats, and therefore want to avoid the performance overhead of more complex type inference?

### Suggestions
**Numeric Preference**
It may be nice if the dev can specify a preferred numerical type to the Read_Csv method. i.e double, long, float, int. I suppose the disadvantage maybe unnecessary precision in some columns, however the default value could still be float. This would probably be the simplest change.

**Enhance Dtypes Parameter**
Alternatively it would be nice to modify the Dtypes[] parameter in Read_Csv to be a dictionary such that we can specify the particular type we want for a given column name. This will make it a little more robust. (I have customer data with the column names in different orders/ shifted in excel etc. It would be nicer to not have to rely on indexing). The disadvantage here is that we would have to check that column names exist etc.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.