apache / apache/arrow

Using cbind when merging multi datasets using open_dataset on a directory.

Open
#33,021 0 comments 0 reactions 0 assignees View on GitHub
Type: task
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

I was wondering if we can use cbind stating particular column names when merging multi datasets using open_dataset(), so that we can bind only those particular cols.

I was using open_dataset to read multi datasets in a particular directory and wanted to merge these multi datasets based on some particular columns that are common to all the datasets.

Is it possible to merge these datasets column wise, since by default open_dataset is merging all the datasets one after the other row-wise?

Do let me know if there's anything like this or any other work around.

**Reporter**: [N Gautam Animesh](https://issues.apache.org/jira/browse/ARROW-17796) / @gautamanimesh

**Note**: *This issue was originally created as [ARROW-17796](https://issues.apache.org/jira/browse/ARROW-17796). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

The issue names open_dataset() and cbind but no source files or tests. Start by locating the open_dataset entry point and reviewing how directory datasets are combined; done means establishing whether selected common columns can be combined column-wise, with the behavior documented and tested.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.