Concurrent Namespace Update Fails with RuntimeException and 500 Error
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 140
Description
### Describe the bug
When performing concurrent parallel HTTP requests involving **UPDATE** operations on namespace properties, an **RuntimeException** is encountered.
The error message in the Polaris logs indicates a **Concurrent modification of namespace** issue:
```
ERROR [org.apa.pol.ser.exc.IcebergExceptionMapper] [,POLARIS] [,,,] (executor-thread-2) Unhandled exception returning INTERNAL_SERVER_ERROR: java.lang.RuntimeException: Concurrent modification of namespace: f62452d6-39fc-40a8-87a5-c08f6ee65c80
at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.setProperties(IcebergCatalog.java:710)
at org.apache.iceberg.rest.CatalogHandlers.updateNamespaceProperties(CatalogHandlers.java:193)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandlerWrapper.updateNamespaceProperties(IcebergCatalogHandlerWrapper.java:538)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.lambda$updateProperties$5(IcebergCatalogAdapter.java:268)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalog(IcebergCatalogAdapter.java:161)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.updateProperties(IcebergCatalogAdapter.java:264)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_ClientProxy.updateProperties(Unknown Source)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi.updateProperties(IcebergRestCatalogApi.java:911)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.updateProperties$$superforward(Unknown Source)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass$$function$$23.apply(Unknown Source)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79)
at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.updateProperties(Unknown Source)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi$quarkusrestinvoker$updateProperties_d0c974b892d05b0179f2423c7b38060f4f880bc3.invoke(Unknown Source)
at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:638)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
```
I’m a bit unsure whether this is actually a bug, but it seems like one. Is it expected that concurrent update requests on the same entity are not supported?
When I tested concurrent requests on catalog, namespace, table, and view entities, I received a CommitFailedException with a 409 status code. I assume that’s the expected behavior. However, I’d like to confirm whether concurrent updates on the same entity (in this case, a namespace) should be allowed or not.
### To Reproduce
Setup a test with a minimum of two threads:
- Ensure that the top-level catalog and a namespace are created before running the test.
- The $n$ amount of threads should **UPDATE** the properties on the namespace in parallel and repeatedly.
Run the test for at least 10 seconds.
During the test, the **UPDATE** operation will occasionally fail with a **500** error. However, it does not fail every time or follow a clear pattern.
This is the endpoint I used to update the properties:
```
http://localhost:8181/api/catalog/v1/polaris/namespaces/test/properties
```
### Actual Behavior
The **UPDATE** operation on the namespace properties fails with a **500** error. The error message indicates a concurrent modification of namespace issue.
### Expected Behavior
If concurrent modifications are not allowed, I would expect a 409 error with a message explaining the conflict. If they are allowed, I’d expect the update to succeed consistently without errors.
### Additional context
Test environment used the in-memory store.
### System information
OS: MacOS Sonoma v15.4.1
Polaris Catalog Version: 1.0.0-incubating-SNAPSHOT
Object Storage: FILE
Contributor guide
Research direction
Reproduce the concurrent requests against /api/catalog/v1/polaris/namespaces/test/properties using the in-memory store, then inspect setProperties in IcebergCatalog.java around line 710 and the REST update path shown in the stack trace. Confirm whether namespace conflicts should map to 409 or whether updates should succeed, and verify the endpoint no longer returns an unexpected 500.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100