[C++] Opening a dataset where partitioning variable is already in the dataset should error differently
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
The error message given when the name of the partition given matches a field in the dataset is a bit misleading - can we catch this earlier and give a different error message?
```r
# R code
library(dplyr)
library(arrow)
tf <- tempfile()
dir.create(tf)
write_dataset(mtcars, tf, partitioning = "cyl", hive_style = FALSE)
# The schema fed into `partitioning` should refer to `cyl` and not `wt`, but the error message doesn't refer to the duplication here
open_dataset(tf, partitioning = schema(wt = int64())) %>% collect()
#> Error in `open_dataset()`:
#> ! Invalid: Unable to merge: Field wt has incompatible types: double vs int64
#> /home/nic2/arrow/cpp/src/arrow/type.cc:1692 fields_[i]->MergeWith(field)
#> /home/nic2/arrow/cpp/src/arrow/type.cc:1755 AddField(field)
#> /home/nic2/arrow/cpp/src/arrow/type.cc:1826 builder.AddSchema(schema)
#> /home/nic2/arrow/cpp/src/arrow/dataset/discovery.cc:262 Inspect(options.inspect_options)
```
**Reporter**: [Nicola Crane](https://issues.apache.org/jira/browse/ARROW-17784) / @thisisnic
**Assignee**: [Sanjiban Sengupta](https://issues.apache.org/jira/browse/ARROW-17784) / @sanjibansg
#### PRs and other links:
- [GitHub Pull Request #14444](https://github.com/apache/arrow/pull/14444)
**Note**: *This issue was originally created as [ARROW-17784](https://issues.apache.org/jira/browse/ARROW-17784). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Reproduce the R example with open_dataset() and the supplied partitioning schema, then read cpp/src/arrow/dataset/discovery.cc and the related merge path in cpp/src/arrow/type.cc. Done means the duplicate partitioning field is detected earlier and produces a distinct, non-misleading error instead of the incompatible-types message; PR #14444 is also linked in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100