Serializable transaction failures are not retried on the server
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Describe the bug
After this [change](https://github.com/apache/polaris/commit/ca38c97111946737ed7df0abcb070d3670c519e6) was introduced, when multiple concurrent requests to create/drop tables or even simply read catalog info are thrown at the webhost, it will quite often throw this:
```
Caused by: java.sql.SQLException: Query failed (#20250319_060102_06461_43kzu): Failed to drop table 'staging_custinvoicejour__2025_03_19_06_01_01_3f698ce2_58ab_4f81_8892_66e014a7a927'
at io.trino.jdbc.ResultUtils.resultsException(ResultUtils.java:33)
at io.trino.jdbc.AsyncResultIterator.lambda$new$1(AsyncResultIterator.java:93)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: zio.Cause$FiberTrace: Exception in thread "zio-fiber-1997042971" java.sql.SQLException: Query failed (#20250319_060102_06461_43kzu): Failed to drop table 'staging_custinvoicejour__2025_03_19_06_01_01_3f698ce2_58ab_4f81_8892_66e014a7a927'
at com.sneaksanddata.arcane.framework.services.merging.JdbcMergeServiceClient.executeBatchQuery(JdbcMergeServiceClient.scala:233)
at com.sneaksanddata.arcane.framework.services.merging.JdbcMergeServiceClient.executeBatchQuery(JdbcMergeServiceClient.scala:234)
at com.sneaksanddata.arcane.framework.services.merging.JdbcMergeServiceClient.executeBatchQuery(JdbcMergeServiceClient.scala:235)
at com.sneaksanddata.arcane.framework.services.streaming.processors.batch_processors.DisposeBatchProcessor.process(DisposeBatchProcessor.scala:25)
at com.sneaksanddata.arcane.framework.services.streaming.processors.batch_processors.DisposeBatchProcessor.process(DisposeBatchProcessor.scala:26)
at com.sneaksanddata.arcane.framework.services.streaming.processors.batch_processors.DisposeBatchProcessor.process(DisposeBatchProcessor.scala:27)
at com.sneaksanddata.arcane.microsoft_synapse_link.services.app.StreamRunnerServiceCdm.run(StreamRunnerServiceCdm.scala:45)
Suppressed: io.trino.jdbc.$internal.client.FailureException: Failed to drop table 'staging_custinvoicejour__2025_03_19_06_01_01_3f698ce2_58ab_4f81_8892_66e014a7a927'
Suppressed: io.trino.jdbc.$internal.client.FailureException: Server error: PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 4.0.5.v202412231137-a96b873527f305f932543045c8679bb1de8d3a43): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: could not serialize access due to read/write dependencies among transactions
Detail: Reason code: Canceled on identification as a pivot, during conflict out checking.
Hint: The transaction might succeed if retried.
Error Code: 0
Call: SELECT CATALOGID, ID, ENTITYVERSION, GRANTRECORDSVERSION, VERSION FROM ENTITIES_CHANGE_TRACKING WHERE ((CATALOGID = ?) AND (ID = ?))
bind => [2 parameters bound]
Query: ReadObjectQuery(referenceClass=ModelEntityChangeTracking sql="SELECT CATALOGID, ID, ENTITYVERSION, GRANTRECORDSVERSION, VERSION FROM ENTITIES_CHANGE_TRACKING WHERE ((CATALOGID = ?) AND (ID = ?))")
Caused by: io.trino.jdbc.$internal.client.FailureException: Failed to drop table 'staging_custinvoicejour__2025_03_19_06_01_01_3f698ce2_58ab_4f81_8892_66e014a7a927'
at io.trino.plugin.iceberg.catalog.rest.TrinoRestCatalog.dropTable(TrinoRestCatalog.java:467)
at io.trino.plugin.iceberg.IcebergMetadata.dropTable(IcebergMetadata.java:2391)
at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.dropTable(ClassLoaderSafeConnectorMetadata.java:452)
at io.trino.tracing.TracingConnectorMetadata.dropTable(TracingConnectorMetadata.java:388)
at io.trino.metadata.MetadataManager.dropTable(MetadataManager.java:1062)
```
### To Reproduce
1. Deploy Polaris 0.9 with Postgres 15.10 metastore backend, with `persistence.xml` shown below:
```
org.eclipse.persistence.jpa.PersistenceProvider
org.apache.polaris.jpa.models.ModelEntity
org.apache.polaris.jpa.models.ModelEntityActive
org.apache.polaris.jpa.models.ModelEntityChangeTracking
org.apache.polaris.jpa.models.ModelEntityDropped
org.apache.polaris.jpa.models.ModelGrantRecord
org.apache.polaris.jpa.models.ModelPrincipalSecrets
org.apache.polaris.jpa.models.ModelSequenceId
NONE
```
2. Deploy Polaris server via Helm chart with **autoscaling enabled**, 2 min replicas, bootstrap from admin tool of the same version as per doc.
3. Throw 30 parallel unique table create/drop/select from statements for 5 minutes, observe the fun
### Actual Behavior
Around 90% of the statements succeed, DROP fail sometimes with the error shown in the issue
### Expected Behavior
No errors thrown
### Additional context
_No response_
### System information
Polaris v0.9, commit from Mar 17, Postgres 15.10 (Aurora), container build deployed on EKS 1.29, from a helm chart built from the same commit.
Contributor guide
Research direction
The report provides a persistence.xml configuration and lists the JPA model classes, but names no Polaris source file or test. Start by reproducing the parallel create/drop/select workload against Polaris 0.9 with PostgreSQL 15.10, then inspect the server persistence path; done means the workload no longer exposes serialization failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, java, kubernetes, postgresql
- Domain
- backend, databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100