apache / apache/polaris

Concurrent Namespace Update Fails with RuntimeException and 500 Error

Open
#1,390 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
2d 1h
Merged PRs (30d)
140

Description

### Describe the bug

When performing concurrent parallel HTTP requests involving **UPDATE** operations on namespace properties, an **RuntimeException** is encountered.

The error message in the Polaris logs indicates a **Concurrent modification of namespace** issue:
```
ERROR [org.apa.pol.ser.exc.IcebergExceptionMapper] [,POLARIS] [,,,] (executor-thread-2) Unhandled exception returning INTERNAL_SERVER_ERROR: java.lang.RuntimeException: Concurrent modification of namespace: f62452d6-39fc-40a8-87a5-c08f6ee65c80
at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.setProperties(IcebergCatalog.java:710)
at org.apache.iceberg.rest.CatalogHandlers.updateNamespaceProperties(CatalogHandlers.java:193)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandlerWrapper.updateNamespaceProperties(IcebergCatalogHandlerWrapper.java:538)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.lambda$updateProperties$5(IcebergCatalogAdapter.java:268)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalog(IcebergCatalogAdapter.java:161)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.updateProperties(IcebergCatalogAdapter.java:264)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_ClientProxy.updateProperties(Unknown Source)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi.updateProperties(IcebergRestCatalogApi.java:911)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.updateProperties$$superforward(Unknown Source)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass$$function$$23.apply(Unknown Source)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79)
at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.updateProperties(Unknown Source)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi$quarkusrestinvoker$updateProperties_d0c974b892d05b0179f2423c7b38060f4f880bc3.invoke(Unknown Source)
at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:638)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
```

I’m a bit unsure whether this is actually a bug, but it seems like one. Is it expected that concurrent update requests on the same entity are not supported?

When I tested concurrent requests on catalog, namespace, table, and view entities, I received a CommitFailedException with a 409 status code. I assume that’s the expected behavior. However, I’d like to confirm whether concurrent updates on the same entity (in this case, a namespace) should be allowed or not.

### To Reproduce

Setup a test with a minimum of two threads:
- Ensure that the top-level catalog and a namespace are created before running the test.
- The $n$ amount of threads should **UPDATE** the properties on the namespace in parallel and repeatedly.

Run the test for at least 10 seconds.

During the test, the **UPDATE** operation will occasionally fail with a **500** error. However, it does not fail every time or follow a clear pattern.

This is the endpoint I used to update the properties:
```
http://localhost:8181/api/catalog/v1/polaris/namespaces/test/properties
```

### Actual Behavior

The **UPDATE** operation on the namespace properties fails with a **500** error. The error message indicates a concurrent modification of namespace issue.

### Expected Behavior

If concurrent modifications are not allowed, I would expect a 409 error with a message explaining the conflict. If they are allowed, I’d expect the update to succeed consistently without errors.

### Additional context

Test environment used the in-memory store.

### System information

OS: MacOS Sonoma v15.4.1
Polaris Catalog Version: 1.0.0-incubating-SNAPSHOT
Object Storage: FILE

Contributor guide

Open the contributing guide

Research direction

Reproduce the concurrent requests against /api/catalog/v1/polaris/namespaces/test/properties using the in-memory store, then inspect setProperties in IcebergCatalog.java around line 710 and the REST update path shown in the stack trace. Confirm whether namespace conflicts should map to 409 or whether updates should succeed, and verify the endpoint no longer returns an unexpected 500.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.