Design a more extensible system of types in fread
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 164
- Avg merge
- 7h 31m
- Merged PRs (30d)
- 1
Description
Current type-handling system in `fread` was designed with specific goal to support type auto-detection across a wide range of types. It was augmented recently to allow disabling certain types (specifically, whether 0/1 should be treated as booleans). Although such system works well for most use cases; it becomes a limiting factor for certain features that we want to implement:
* Allow the user to request a specific "stype" for a column, such as `float32`. This doesn't pinpoint the exact "ctype", since there could be multiple parsers for `float32`s, however this shouldn't really be a user's concern... (see #626)
* Allow the user to request a specific "ltype", such as `int`. Similar to above, but without specifying the storage size. This is needed in particular for Arff readers (see #511).
* Create separate `void` type for a column that matches NAs only and can be seen as matching any other parse type. This is needed to improve type-detection of column names (see #656), and detect the case of extra-void column (i.e. when there is an extra separator at the end of each line).
* Allow the user to choose what should happen if a value cannot be read with the requested parser. Possible actions could be: **"skip"** (replace with NA), **"error"** (throw an exception immediately), **"report"** (fill with NA, but also collect the information about the bad field into a separate table, and give it to the user in the end), **"bump"** (bump the entire column to a higher type, current behavior).
* Allow the user to choose what should happen if the value is too large to fit in the specified precision (e.g. a value doesn't fit into int32). Possible actions could be: **"bump"** (use larger type, current behavior), **"coerce"** (as if `static_cast` is applied), **"clamp"** (replace with largest possible value), **"error"** (throw an exception), **"report"** (fill with coerced value, and collect information about the failed field into a separate table).
* Allow the user to request a type incompatible with the autodetected one. Because "the user is always right". For example, if a column contains numbers with occasional strings, then the user may want to keep the column as numeric discarding all non-numbers.
* Support more complex type hierarchies than simple linear ordering.
----
The proposed solution is to have clear separation between "requested types" and "resolved types" (parsers). The **"requested type"** is the column type as requested by the user: `drop`, `auto`, `bool`, `int`, `real`, `time`, `string`, `int32`, `int64`, `float32`, `float64`, etc. Each of the requested types maps into 1 or more **"resolved type"**'s, and each resolved type has the associated parser, together with some meta info: parser's name, code, output size, etc.
Even seemingly unambiguous types such as `int32` may internally be served by multiple parsers: plain, with-whitespace-or-quotes, with-thousand-separators, hex, binary, etc. Not all of these internal parsers will be exposed to the user.
In the future there will also be a need to add customizable parse types. These could be: `datetime` type with custom user-supplied (or auto-detected) format; `decimal` type, with the number of decimal digits as the parameter; `regexp`-powered parsing, if we ever run out of other things to do; etc. These can be either dynamically added to the array of parsers (if there are not too many such columns in the file), or designated a special value in the types array, with separate logic to handle those types of fields.
cc: @mattdowle
Contributor guide
Assessment
This issue has not been assessed yet.