apache / apache/arrow

[C++] Opening a dataset where partitioning variable is already in the dataset should error differently

Open
#33,011 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

The error message given when the name of the partition given matches a field in the dataset is a bit misleading - can we catch this earlier and give a different error message?
```r

# R code
library(dplyr)
library(arrow)

tf <- tempfile()
dir.create(tf)
write_dataset(mtcars, tf, partitioning = "cyl", hive_style = FALSE)
# The schema fed into `partitioning` should refer to `cyl` and not `wt`, but the error message doesn't refer to the duplication here
open_dataset(tf, partitioning = schema(wt = int64())) %>% collect()
#> Error in `open_dataset()`:
#> ! Invalid: Unable to merge: Field wt has incompatible types: double vs int64
#> /home/nic2/arrow/cpp/src/arrow/type.cc:1692 fields_[i]->MergeWith(field)
#> /home/nic2/arrow/cpp/src/arrow/type.cc:1755 AddField(field)
#> /home/nic2/arrow/cpp/src/arrow/type.cc:1826 builder.AddSchema(schema)
#> /home/nic2/arrow/cpp/src/arrow/dataset/discovery.cc:262 Inspect(options.inspect_options)
```

**Reporter**: [Nicola Crane](https://issues.apache.org/jira/browse/ARROW-17784) / @thisisnic
**Assignee**: [Sanjiban Sengupta](https://issues.apache.org/jira/browse/ARROW-17784) / @sanjibansg
#### PRs and other links:
- [GitHub Pull Request #14444](https://github.com/apache/arrow/pull/14444)

**Note**: *This issue was originally created as [ARROW-17784](https://issues.apache.org/jira/browse/ARROW-17784). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the R example with open_dataset() and the supplied partitioning schema, then read cpp/src/arrow/dataset/discovery.cc and the related merge path in cpp/src/arrow/type.cc. Done means the duplicate partitioning field is detected earlier and produces a distinct, non-misleading error instead of the incompatible-types message; PR #14444 is also linked in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.