apache / apache/iceberg

GCSFileIO - how to connection pooling - exploding number of TCP connections

Open
#15,411 8 comments 0 reactions 0 assignees View on GitHub
improvement stale
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

At the moment, we don't see any indications TCP connection pooling is possible with GCSFileIO. This causes a massive spike of TCP connections. Like literally every 15 mins it spawns thousands of TCP conns, just to MERGE a few partitions...

### Query engine

Spark 4

### Willingness to contribute

- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.