apache / apache/polaris

Deleting Large Tables causes the system to exhaust connections pool or become unresponsive.

Open
#3,462 8 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Describe the bug

I have a very large table that was created through CDC. After executing DROP TABLE from trino, the whole polaris instance becomes unresponsive with logs like below:

Running polaris as a standalone (not docker).
```
2026-01-16 16:42:59,043 WARN [org.apa.pol.ser.tas.FileCleanupTaskHandler] [,] [,,,] () file=s3://testdec16/allo_custom/dbo__cdc_test/data/00001-1768518467576-0d5953a4-ceed-49ed-a763-6ebb7b66a939-00001.parquet tableIdentifier=allo_custom.dbo__cdc_test baseFile=s3://testdec16/allo_custom/dbo__cdc_test/metadata/58be18c0-aa21-4b89-84dd-e1eb953f0444-m2.avro Exception caught deleting data file
2026-01-16 16:42:59,043 WARN [org.apa.pol.ser.tas.FileCleanupTaskHandler] [,] [,,,] () file=s3://testdec16/allo_custom/dbo__cdc_test/data/00001-1768518467576-0d5953a4-ceed-49ed-a763-6ebb7b66a939-00001.parquet attempt=2 error=java.lang.RuntimeException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool (SDK Attempt Count: 1) Error encountered attempting to delete file
2026-01-16 16:42:59,043 WARN [org.apa.pol.ser.tas.FileCleanupTaskHandler] [,] [,,,] () file=s3://testdec16/allo_custom/dbo__cdc_test/data/00001-1768518467292-00defaac-2afc-4248-959c-f08f9294f161-00001.parquet tableIdentifier=allo_custom.dbo__cdc_test baseFile=s3://testdec16/allo_custom/dbo__cdc_test/metadata/58be18c0-aa21-4b89-84dd-e1eb953f0444-m2.avro Exception caught deleting data file
2026-01-16 16:42:59,043 WARN [org.apa.pol.ser.tas.FileCleanupTaskHandler] [,] [,,,] () file=s3://testdec16/allo_custom/dbo__cdc_test/data/00001-1768518467292-00defaac-2afc-4248-959c-f08f9294f161-00001.parquet attempt=2 error=java.lang.RuntimeException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool (SDK Attempt Count: 1) Error encountered attempting to delete file
```

After a while, it starts to delete them eventually, but during this process, you cannot access catalog, list tables, create table or query data.

Using

postgresql as a JDBC backend.

Here is my application config for using On-Prem S3.

```
polaris.realm-context.realms=polaris
quarkus.log.console.level=DEBUG
quarkus.log.category."software.amazon.awssdk.request".level=DEBUG
quarkus.log.category."software.amazon.awssdk.auth.signer".level=DEBUG
quarkus.log.category."software.amazon.awssdk.http".level=DEBUG
quarkus.log.category."io.smallrye.config".level=DEBUG
polaris.realm-context.header-name=Polaris-Realm
#polaris.realm-context.require-header=true
polaris.persistence.type=relational-jdbc
quarkus.datasource.db-kind=postgresql
quarkus.datasource.username=user_name
quarkus.datasource.password=password
quarkus.datasource.jdbc.url=jdbc:postgresql://postgresql:5432/polaris_test
polaris.authentication.token-broker.type=rsa-key-pair
polaris.authentication.token-broker.rsa-key-pair.public-key-file=/opt/polaris/public.key
polaris.authentication.token-broker.rsa-key-pair.private-key-file=/opt/polaris/private.key
polaris.authentication.token-broker.max-token-generation=PT1H
polaris.bootstrap.credentials=polaris,admin,creds
polaris.storage.aws.access-key=access_key
polaris.storage.aws.secret-key=secret_key
polaris.catalog.s3.no-sts=true
#polaris.catalog.s3.region=us-east-1
#polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION": true
polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES": ["S3"]
polaris.features."ALLOW_SETTING_S3_ENDPOINTS"=true
polaris.features."DROP_WITH_PURGE_ENABLED"=true
polaris.features."CLEANUP_ON_CATALOG_DROP"=true
```

### To Reproduce

_No response_

### Actual Behavior

_No response_

### Expected Behavior

_No response_

### Additional context

_No response_

### System information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing DROP TABLE with a large CDC-created table using the PostgreSQL backend and S3 configuration shown, then inspect FileCleanupTaskHandler warnings and connection-pool behavior. Done means large-table deletion completes without exhausting the pool and catalog operations remain responsive throughout.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, postgresql
Domain
backend, cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.