Permanent worker-thread hang on node update with micronode-list field

Hello everyone

## Summary

Updating a node that contains a **micronode-list field** can **permanently block a Vert.x worker thread** when the underlying persistence throws (e.g. a MariaDB deadlock). The exception is swallowed by an RxJava 2 anti-pattern, so the `blockingGet()` in `RestUpdaters.MICRONODE_LIST_UPDATER` never returns. The thread is lost for good; repeated occurrences exhaust the worker pool and the instance stops serving requests.

We observed worker threads blocked for **10–14 hours** in production (a permanent hang, not slowness). This became frequent after migrating to the v3 Hibernate/MariaDB backend, which raises real SQL exceptions where the legacy graph backend did not.

## Environment

- Gentics Mesh **3.2.x** (Hibernate/MariaDB backend)
- Vert.x 5.x, RxJava **2.2.21**
- MariaDB 12.x

## Symptoms

`vertx-blocked-thread-checker` reports threads blocked far beyond the limit, always on the same stack:

```
WARN [vertx-blocked-thread-checker] Thread vert.x-worker-thread-19 has been blocked for 36885865 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
at java.util.concurrent.CountDownLatch.await(...)
at io.reactivex.Single.blockingGet(Single.java:2870)
at ...RestUpdaters.lambda$static$11(RestUpdaters.java:509) // MICRONODE_LIST_UPDATER
at ...HibUnmanagedFieldContainer.updateFieldsFromRest(...)
at ...PersistingNodeDao.update(...)
at ...NodeEndpoint.lambda$addUpdateHandler$23(NodeEndpoint.java:590)
```

The thread is parked on `Single.blockingGet` → `CountDownLatch.await`, i.e. it is waiting for a reactive result that will never arrive.

## Root cause

`HibMicronodeFieldList.update(InternalActionContext, MicronodeFieldList)` uses `Observable.create(...)` and subscribes with a `(onSuccess, onError)` pair. The persistence writes (`removeAll()`, `insertReferenced()`, `deleteReferenced()`) run **inside the `onSuccess` consumer**:

```java
return Observable.create(subscriber -> {
...
...toList().subscribe(micronodeList -> { // onSuccess consumer
removeAll(); // Hibernate write — can throw
for (HibMicronode m : micronodeList) {
insertReferenced(counter++, m); // Hibernate write — can throw
}
existing.values().forEach(m -> deleteReferenced(m)); // Hibernate write — can throw
subscriber.onNext(true);
subscriber.onComplete(); // reached ONLY if nothing above threw
}, e -> {
subscriber.onError(e); // does NOT catch throws from the onSuccess consumer
});
}).singleOrError();
```

Called from `RestUpdaters.MICRONODE_LIST_UPDATER`:

```java
// RestUpdaters.java (~509 in 3.2.1)
// TODO instead this method should also return an observable
micronodeGraphFieldList.update(ac, micronodeList).blockingGet();
```

**The RxJava 2 trap:** if the `onSuccess` consumer (`micronodeList -> { … }`) throws, the exception is **not** delivered to the sibling `onError` consumer. It is routed to the global `RxJavaPlugins.onError` handler (as an `UndeliverableException`) and effectively lost. Consequently:

1. `removeAll()` / `insertReferenced()` / `deleteReferenced()` throw a Hibernate exception.
2. `subscriber.onNext()` / `onComplete()` are never called.
3. The outer `Observable.create` never terminates.
4. `singleOrError().blockingGet()` waits forever on its `CountDownLatch`.
5. The worker thread is lost — without even returning the expected HTTP 500.

The existing `// TODO instead this method should also return an observable` shows this blocking design is already known to be problematic.

## Why the v3 backend exposes it

With the Hibernate/MariaDB backend, persisting micronodes (`removeAll` / `insert` / `delete`) now hits a real SQL database and can throw (constraint violations, **lock-wait timeout**, **deadlocks**, optimistic locking). In our case the trigger is a MariaDB deadlock:

```
Error 1213 (SQLState 40001): Deadlock found when trying to get lock; try restarting transaction
SQL: INSERT INTO mesh_stringlistitem (...) // string-list items inside the micronodes
```

The swallow-and-hang bug already existed; the v3 backend just makes it fire in practice.

## Two failure modes (same underlying exception)

Depending on where the exception is thrown inside `update()`, behaviour differs — distinguishable by the `blockingGet` line:

| Where the exception is thrown | RxJava routing | `blockingGet` | Result |
|---|---|---|---|
| Inside the `flatMap` mapper (`micronode.updateFieldsFromRest(...)`) | correctly routed to `onError` | `Single.java:2869` | clean HTTP 500, thread released |
| Inside the `onSuccess` consumer (`removeAll`/`insert`/`delete`) | **swallowed** | `Single.java:2870` | **thread hangs forever** |

The first mode is a working stack (exception propagates); the second is the permanent hang described above.

## Minimal reproduction

The hang is purely an RxJava issue and reproduces without Mesh or a database, using the same RxJava version (2.2.21):

```java
Single update = Observable.create(subscriber -> {
Observable.fromIterable(List.of("a", "b"))
.toList()
.subscribe(list -> {
throw new RuntimeException("simulated Hibernate deadlock"); // onSuccess throws
// subscriber.onNext(true); subscriber.onComplete(); // never reached
}, e -> subscriber.onError(e)); // never receives it
}).singleOrError();

update.blockingGet(); // blocks forever on CountDownLatch.await (Single.java:2870)
```

## Impact

- **Severity: high.** Each occurrence permanently loses a worker thread (and holds its DB connection). Repeated occurrences exhaust the worker pool and the HikariCP pool → the instance stops serving requests.
- Not detectable by a standard `livenessProbe` on `/health/live`: that check runs on the event loop, which stays responsive while all worker threads are dead.
- Affects any node update touching a micronode-list field whenever persistence throws.

Start with HibMicronodeFieldList.update and RestUpdaters.java around line 509, then run the minimal RxJava 2.2.21 reproduction described in the issue. Trace how exceptions from the persistence calls reach the observable and verify that a failing update terminates with an error and releases blockingGet instead of hanging indefinitely.

Written by the indexing model from the issue text.