apache / apache/polaris

Concurrent CREATE and DELETE Operations Cause IllegalStateException

Open
#1,389 4 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Describe the bug

When performing concurrent parallel HTTP requests involving **CREATE** and **DELETE** operations on namespace, table, and view entities, an **IllegalStateException** is encountered.

Polaris logs the following error message:
```
ERROR [org.apa.pol.ser.exc.IcebergExceptionMapper] [,POLARIS] [,,,] (executor-thread-83) Unhandled exception returning INTERNAL_SERVER_ERROR: java.lang.IllegalStateException: Unable to resolve sibling entities to validate location - PATH_COULD_NOT_BE_FULLY_RESOLVED
at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.validateNoLocationOverlap(IcebergCatalog.java:1121)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.createNamespaceInternal(IcebergCatalog.java:524)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalog.createNamespace(IcebergCatalog.java:500)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandlerWrapper.createNamespace(IcebergCatalogHandlerWrapper.java:485)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.lambda$createNamespace$0(IcebergCatalogAdapter.java:198)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalog(IcebergCatalogAdapter.java:161)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.createNamespace(IcebergCatalogAdapter.java:195)
at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_ClientProxy.createNamespace(Unknown Source)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi.createNamespace(IcebergRestCatalogApi.java:166)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.createNamespace$$superforward(Unknown Source)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass$$function$$2.apply(Unknown Source)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79)
at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.createNamespace(Unknown Source)
at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi$quarkusrestinvoker$createNamespace_6b5544448686388164ce7bf7f7e9768cc6a8f333.invoke(Unknown Source)
at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:638)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)

```

Http reposonse body:
```
{
"error":
{
"message":"Unable to resolve sibling entities to validate location - PATH_COULD_NOT_BE_FULLY_RESOLVED",
"type":"IllegalStateException",
"code":500
}
}
```

### To Reproduce

Setup a test with a minimum of two threads:
- Ensure that the top-level catalog and a namespace are created before running the test. And make sure the principal has the required permissions to create and delete the entities.
- The $n$ amount of threads should **CREATE** and **DELETE** the aforementioned entities in parallel and repeatedly. The thread should first create an entity and then immediately delete it.

Run the test for at a least 10 seconds.

During the test, the **CREATE** operation will occasionally fail with a **500** error. However, it does not fail every time or follow a specific pattern. The **DELETE** operation seems to work fine.

These are the endpoints I used for the test, with the host and resource being `http://localhost:8181/api/catalog/v1`:
- **Create Namespace**: `POST /{prefix}/namespaces`
- **Delete Namespace**: `DELETE /{prefix}/namespaces/{namespace}`
- **Create Table**: `POST /{prefix}/namespaces/{namespace}/tables`
- **Delete Table**: `DELETE /{prefix}/namespaces/{namespace}/tables/{table}`
- **Create View**: `POST /{prefix}/namespaces/{namespace}/views`
- **Delete View**: `DELETE /{prefix}/namespaces/{namespace}/views/{view}`

### Actual Behavior

The **CREATE** operation occasionally fails with a **500** error with a **IllegalStateException** in the logs.

### Expected Behavior

I would expect the **CREATE** operation to succeed even if there are concurrent operations happening on the top-level entity.

### Additional context

Test environment used the in-memory store.

### System information

OS: MacOS Sonoma v15.4.1
Polaris Catalog Version: 1.0.0-incubating-SNAPSHOT
Object Storage: FILE

Contributor guide

Open the contributing guide

Research direction

Start with IcebergCatalog.java, especially validateNoLocationOverlap around line 1121 and the createNamespaceInternal path around line 524. Reproduce the issue using at least two threads against the listed create and delete endpoints for 10 seconds with the in-memory store. Done means repeated concurrent CREATE and DELETE operations complete without intermittent 500 IllegalStateException responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.