googleapis / googleapis/google-cloud-java

[java-spanner] Spanner: AsyncRunner.runAsync blocks executor; deadlock possible

未关闭
#12,259 3 条评论 2 个 reaction 已指派 1 人 已指派给 @olavloite 在 GitHub 查看
api: spanner
主要语言
Java
星标
2.1k
派生
1.2k
平均合并
1 天 23 小时
30 天内合并 PR
154

描述

`AsyncRunner.runAsync` performs a [blocking get](https://github.com/googleapis/java-spanner/blob/v6.52.1/google-cloud-spanner/src/main/java/com/google/cloud/spanner/AsyncRunnerImpl.java#L60) inside its executor. This may result in deadlock if the following conditions are met:
* The executor is a thread pool with a maximum size.
* The same thread pool is also used by the work itself.
* The number of concurrent transactions is equal to or greater than the maximum number of threads.

This is most likely to manifest itself if the application has a constrained global thread pool optimized for running non-blocking operations. In particular, we noticed this in a Cats Effect application, although the bug is not restricted to Scala or Cats Effect. ~~This may also be causing the performance problems noted in googleapis/java-spanner#1751.~~

Our workaround is to provide a dedicated thread pool that is only used for `runAsync`, but which is not used by the work itself.

I've built a [minimum test case](https://github.com/zobar/ubiquitous-garbanzo/blob/main/app/src/main/java/com/jackhenry/dkleinschmidt/App.java) in Java to demonstrate this issue.

#### Environment details

**OS type and version:** Mac OS Ventura 13.6
**Java version:** OpenJDK 21
**Versions:** `com.google.cloud:google-cloud-spanner:6.52.1`

#### Steps to reproduce

1. Clone [my GitHub repo](https://github.com/zobar/ubiquitous-garbanzo) for a minimized test case.
2. Run 15 concurrent transactions on a maximum of 16 threads. This will work as expected.
```
$ ./gradlew run --args="16 15 my-project my-instance my-database"
Getting results...
Successfully ran 15 transactions in parallel on 16 threads.
$
```
2. Try running 16 concurrent transactions on a maximum of 16 threads. This will hang.
```
$ ./gradlew run --args="16 16 my-project my-instance my-database"
Getting results...
^C
```

#### Code example

```java
public int run() throws ExecutionException, InterruptedException {
SettableApiFuture trigger = SettableApiFuture.create();

Iterable transactions = Stream
.generate(() -> databaseClient
.runAsync()
.runAsync(work(trigger), threadPool))
.limit(concurrency)
.toList();
ApiFuture results = ApiFutures.allAsList(transactions);

trigger.set(null);

System.out.println("Getting results...");
return results.get().size();
}

public AsyncRunner.AsyncWork work(ApiFuture trigger) {
return (txn) -> ApiFutures.transform(trigger, (input) -> input, threadPool);
}
```

#### External references such as API reference guides

The [API documentation](https://cloud.google.com/java/docs/reference/google-cloud-spanner/latest/com.google.cloud.spanner.AsyncRunner#com_google_cloud_spanner_AsyncRunner__R_runAsync_com_google_cloud_spanner_AsyncRunner_AsyncWork_R__java_util_concurrent_Executor_) for `runAsync` does not mention that it runs blocking operations in the executor.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。