apache / apache/arrow

[C++] Allow filesystems to be implemented in separate library (+ move remote filesystems out of libarrow into their own shared libraries)

Open
#49,399 0 comments 2 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

This is a follow up of the original issue with the same title which shouldn't have been closed originally:
- https://github.com/apache/arrow/issues/38309

This issue should be an issue covering different sub-issues to track the overall goal. Please do not close this issue for interim steps and keep it to cover the full feature.

Currently all our supported file systems in C++ (local, hdfs, s3, google cloud (gs/gcs), azure (abfs)) are incorporated in the core libarrow library.

There is a desire to be able to separate those filesystems in their own libraries, such that they can be installed separately. The remote filesystems each come with their own (potentially quite large) dependencies, and one typically doesn't need all of them at the same time.

There has been work towards achieving this goal in the past:
- Creating a FileSystem Registry and factory where users can add their own filesystems (https://github.com/apache/arrow/pull/39067)
- Moving the Local Filesystem to the registry: https://github.com/apache/arrow/issues/40342
- Allowing to move s3 to the registry as it's own module (even though this does not contain the same functionality as s3 built as part of libarrow): https://github.com/apache/arrow/issues/40343

In order to move all the functionality for S3 to be part of it's own module see the following issue and discussion
- https://github.com/apache/arrow/issues/46369

This is also in-line with some work that we've done in order to break down libarrow.so into smaller submodules like:
- https://github.com/apache/arrow/pull/46261

Related issue on conda feedstocks that spawned the creation of this one:
- https://github.com/conda-forge/arrow-cpp-feedstock/issues/944

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.