Using cbind when merging multi datasets using open_dataset on a directory.
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
I was wondering if we can use cbind stating particular column names when merging multi datasets using open_dataset(), so that we can bind only those particular cols.
I was using open_dataset to read multi datasets in a particular directory and wanted to merge these multi datasets based on some particular columns that are common to all the datasets.
Is it possible to merge these datasets column wise, since by default open_dataset is merging all the datasets one after the other row-wise?
Do let me know if there's anything like this or any other work around.
**Reporter**: [N Gautam Animesh](https://issues.apache.org/jira/browse/ARROW-17796) / @gautamanimesh
**Note**: *This issue was originally created as [ARROW-17796](https://issues.apache.org/jira/browse/ARROW-17796). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
The issue names open_dataset() and cbind but no source files or tests. Start by locating the open_dataset entry point and reviewing how directory datasets are combined; done means establishing whether selected common columns can be combined column-wise, with the behavior documented and tested.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100