GCSFileIO - how to connection pooling - exploding number of TCP connections
Open
improvement
stale
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
At the moment, we don't see any indications TCP connection pooling is possible with GCSFileIO. This causes a massive spike of TCP connections. Like literally every 15 mins it spawns thousands of TCP conns, just to MERGE a few partitions...
### Query engine
Spark 4
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Assessment
This issue has not been assessed yet.