apache / apache/arrow

[C++] Add GCS connection pool size option

Open
#20,314 4 comments 0 reactions 1 assignee Claimed by @azhu248 View on GitHub
Component: C++ good-first-issue Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Multi-threaded read performance in Arrow's GCS file system implementation currently is relatively low. Given the high latency of cloud blob systems like GCS, a common strategy is to use many concurrent readers (if the system has enough memory to support that), e.g. using 100 threads.

The GCS client library offers a [`ConnectionPoolSize` option](https://googleapis.dev/cpp/google-cloud-storage/latest/structgoogle_1_1cloud_1_1storage_1_1v1_1_1ConnectionPoolSizeOption.html). If this option is set to a value that's too low, concurrency is throttled. At the moment, this is not exposed in [`GcsOptions`](https://github.com/apache/arrow/blob/73cdd6a59b52781cc43e097ccd63ac36f705ee2e/cpp/src/arrow/filesystem/gcsfs.h#L59), consequently limiting multi-threaded throughput.

Instead of exposing this option, an alternative implementation strategy could be to use the same value as set by `arrow::io::SetIOThreadPoolCapacity`.

**Reporter**: [Leonhard Gruenschloss](https://issues.apache.org/jira/browse/ARROW-17033)

**Note**: *This issue was originally created as [ARROW-17033](https://issues.apache.org/jira/browse/ARROW-17033). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.