Cleanup tasks failing with value too long for type character varying(255)
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Describe the bug
When a table is dropped using DROP_WITH_PURGE (default in Trino connector), Polaris fails to drop the table data from the underlying storage, throwing this:
```
2025-03-18 12:46:24,760 INFO [org.apa.pol.ser.tas.TaskExecutorImpl] [,] [00000000000000000000000000000000,,0000000000000000,false] (executor-thread-1) Handling task entity id 155
2025-03-18 12:46:24,855 INFO [org.apa.pol.ser.tas.TableCleanupTaskHandler] [,] [00000000000000000000000000000000,,0000000000000000,false] (executor-thread-1) tableIdentifier=d365_eu.test metadataLocation=s3://.../metadata/00001-a8283615-e997-490a-8c48-28d75a02f29b.metadata.json Handling table metadata cleanup task
2025-03-18 12:46:24,856 INFO [org.apa.ice.CatalogUtil] [,] [00000000000000000000000000000000,,0000000000000000,false] (executor-thread-1) Loading custom FileIO implementation: org.apache.iceberg.aws.s3.S3FileIO
[EL Warning]: 2025-03-18 12:46:24.944--UnitOfWork(169674653)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 4.0.5.v202412231137-a96b873527f305f932543045c8679bb1de8d3a43): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: value too long for type character varying(255)
Error Code: 0
Call: INSERT INTO ENTITIES (CATALOGID, ID, CREATETIMESTAMP, DROPTIMESTAMP, ENTITYVERSION, GRANTRECORDSVERSION, INTERNALPROPERTIES, LASTUPDATETIMESTAMP, NAME, PARENTID, PROPERTIES, PURGETIMESTAMP, SUBTYPECODE, TOPURGETIMESTAMP, TYPECODE, VERSION) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
bind => [16 parameters bound]
Query: InsertObjectQuery(org.apache.polaris.jpa.models.ModelEntity@ee2670d)
2025-03-18 12:46:26,945 WARN [org.apa.pol.ser.tas.TaskExecutorImpl] [,] [,,,] (executor-thread-1) Failed to handle task entity id 155: java.util.concurrent.CompletionException: jakarta.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 4.0.5.v202412231137-a96b873527f305f932543045c8679bb1de8d3a43): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: value too long for type character varying(255)
Error Code: 0
Call: INSERT INTO ENTITIES (CATALOGID, ID, CREATETIMESTAMP, DROPTIMESTAMP, ENTITYVERSION, GRANTRECORDSVERSION, INTERNALPROPERTIES, LASTUPDATETIMESTAMP, NAME, PARENTID, PROPERTIES, PURGETIMESTAMP, SUBTYPECODE, TOPURGETIMESTAMP, TYPECODE, VERSION) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
bind => [16 parameters bound]
Query: InsertObjectQuery(org.apache.polaris.jpa.models.ModelEntity@ee2670d)
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
at io.smallrye.context.impl.wrappers.SlowContextualRunnable.run(SlowContextualRunnable.java:19)
at org.jboss.threads.EnhancedViewExecutor$EnhancedViewExecutorRunnable.run(EnhancedViewExecutor.java:501)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:643)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: jakarta.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 4.0.5.v202412231137-a96b873527f305f932543045c8679bb1de8d3a43): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: value too long for type character varying(255)
Error Code: 0
```
### To Reproduce
1. Have a working setup with Trino using Polaris as Iceberg REST catalog, S3 as a backing store, and make sure principal used has both TABLE_DROP and TABLE_WRITE_DATA permissions
2. Create a table from Trino: `create table mycatalog.myschema.test as select 'a' as colA`
3. Confirm table creation by running `select * from mycatalog.myschema.test`
4. Drop the table: `drop table mycatalog.myschema.test`
### Actual Behavior
Table is removed from catalog, but data is not purged from S3, error printed to logs
### Expected Behavior
Data is purged
### Additional context
This can be fixed by manually applying `text` data type to tables:
```sql
ALTER TABLE entities
ALTER COLUMN internalproperties SET DATA TYPE text,
ALTER COLUMN name SET DATA TYPE text,
ALTER COLUMN properties SET DATA TYPE text;
```
and
```
ALTER TABLE entities_active
ALTER COLUMN name SET DATA TYPE text;
```
```
2025-03-18 13:38:01,000 INFO [org.apa.pol.ser.tas.TaskExecutorImpl] [,polaris] [00000000000000000000000000000000,,0000000000000000,false] (executor-thread-2) Handling task entity id 204
2025-03-18 13:38:01,000 INFO [org.apa.pol.ser.tas.TaskExecutorImpl] [,polaris] [00000000000000000000000000000000,,0000000000000000,false] (executor-thread-3) Handling task entity id 205
2025-03-18 13:38:01,017 INFO [org.apa.pol.ser.tas.TaskExecutorImpl] [,polaris] [00000000000000000000000000000000,,0000000000000000,false] (executor-thread-1) taskEntityId=202 handlerClass=class org.apache.polaris.service.task.TableCleanupTaskHandler Task successfully handled
2025-03-18 13:38:01,034 INFO [org.apa.ice.CatalogUtil] [,polaris] [00000000000000000000000000000000,,0000000000000000,false] (executor-thread-2) Loading custom FileIO implementation: org.apache.iceberg.aws.s3.S3FileIO
2025-03-18 13:38:01,105 INFO [org.apa.ice.CatalogUtil] [,polaris] [00000000000000000000000000000000,,0000000000000000,false] (executor-thread-3) Loading custom FileIO implementation: org.apache.iceberg.aws.s3.S3FileIO
2025-03-18 13:38:01,405 INFO [org.apa.pol.ser.tas.ManifestFileCleanupTaskHandler] [,] [,,,] () manifestFile=s3://.../test/metadata/4f0fd778-9c8c-4ff6-8b9f-440292036a3f-m0.avro All data files in manifest deleted - deleting manifest
2025-03-18 13:38:01,413 INFO [org.apa.pol.ser.tas.TaskExecutorImpl] [,polaris] [,,,] (executor-thread-3) taskEntityId=205 handlerClass=class org.apache.polaris.service.task.BatchFileCleanupTaskHandler Task successfully handled
2025-03-18 13:38:01,421 INFO [org.apa.pol.ser.tas.TaskExecutorImpl] [,polaris] [,,,] (executor-thread-2) taskEntityId=204 handlerClass=class org.apache.polaris.service.task.ManifestFileCleanupTaskHandler Task successfully handled
```
### System information
Polaris v0.9 with commit from Mar 17, built with Postgres Eclipse Link for container. Postgres database is Aurora Serverless 15.10, bootstrapped with Admin tool of the same version.
`./gradlew clean :polaris-quarkus-server:assemble -Dquarkus.container-image.build=true --no-build-cache -PeclipseLink=true -PeclipseLinkDeps=org.postgresql:postgresql:42.7.4`
`./gradlew clean :polaris-quarkus-admin:build -Dquarkus.container-image.build=true --no-build-cache -PeclipseLinkDeps=org.postgresql:postgresql:42.7.4`
Installed in EKS 1.29 via Helm chart built from the same commit.
Contributor guide
Research direction
Start by reproducing the Trino DROP TABLE flow against the PostgreSQL-backed catalog, then trace the schema definitions for ENTITIES and ENTITIES_ACTIVE and the cleanup task path shown in the logs. Compare the affected columns with the stored values and verify that cleanup completes without the length error and that the table data is purged from S3.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, postgresql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100