apache / apache/arrow

[C++] unify_schemas can't handle int64 + double, affects CSV dataset

Open
#30,245 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Twitter question of "how can I make arrow's csv reader not make int64 for integers", turns out to be originating from the scenario where some csvs in a directory may have all integer values for a column but there are decimals in others, and you can't use them together in a dataset.
```r

library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)

ds_dir <- tempfile()
dir.create(ds_dir)
cat("a\n1", file = file.path(ds_dir, "1.csv"))
cat("a\n1.1", file = file.path(ds_dir, "2.csv"))

ds <- open_dataset(ds_dir, format = "csv")
ds
#> FileSystemDataset with 2 csv files
#> a: int64

## It just picked the schema of the first file
collect(ds)
#> Error: Invalid: Could not open CSV input source '/private/var/folders/yv/b6mwztyj0r11r8pnsbmpltx00000gn/T/RtmpzENOMb/filea9c3292e06dd/2.csv': Invalid: In CSV column #0: Row #2: CSV conversion error to int64: invalid value '1.1'
#> ../src/arrow/csv/converter.cc:492 decoder_.Decode(data, size, quoted, &value)
#> ../src/arrow/csv/parser.h:123 status
#> ../src/arrow/csv/converter.cc:496 parser.VisitColumn(col_index, visit)
#> ../src/arrow/csv/reader.cc:462 internal::UnwrapOrRaise(maybe_decoded_arrays)
#> ../src/arrow/compute/exec/exec_plan.cc:398 iterator_.Next()
#> ../src/arrow/record_batch.cc:318 ReadNext(&batch)
#> ../src/arrow/record_batch.cc:329 ReadAll(&batches)

## Let's try again and tell it to unify schemas. Should result in a float64 type
ds <- open_dataset(ds_dir, format = "csv", unify_schemas = TRUE)
#> Error: Invalid: Unable to merge: Field a has incompatible types: int64 vs double
#> ../src/arrow/type.cc:1621 fields_[i]->MergeWith(field)
#> ../src/arrow/type.cc:1684 AddField(field)
#> ../src/arrow/type.cc:1755 builder.AddSchema(schema)
#> ../src/arrow/dataset/discovery.cc:251 Inspect(options.inspect_options)
```

**Reporter**: [Neal Richardson](https://issues.apache.org/jira/browse/ARROW-14705) / @nealrichardson
#### Related issues:
- [[C++] allow unify schema to coalesce int64 and float64 ](https://github.com/apache/arrow/issues/30235) (duplicates)
- [[R] Add option to attempt 32-bit integer type inference in CSV reader](https://github.com/apache/arrow/issues/30083) (relates to)
#### PRs and other links:
- [GitHub Pull Request #12000](https://github.com/apache/arrow/pull/12000)

**Note**: *This issue was originally created as [ARROW-14705](https://issues.apache.org/jira/browse/ARROW-14705). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

The failure path points to src/arrow/type.cc, where schema fields are merged; start by reading related issue #30235 and PR #12000. Confirm the existing CSV dataset reproduction, then verify that schemas containing int64 and double can be unified and collected as a compatible floating-point dataset.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, r
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.