Watermark can become unavailable for executors while it's updated with new values

The watermark is updated by the driver like so:

```

blockManager.removeBlock(WATERMARKS_BLOCK_ID, true);
blockManager.putSingle(WATERMARKS_BLOCK_ID, newValues,
StorageLevel.MEMORY_ONLY(),
true);

```

However, these operations are neither synchronous nor atomic, so if an executor requests the watermark values before they are actually put but after they have been removed, it may get a `null` as a response, which will make it default to negative infinity as the watermark. This can result in an erroneous results.

To overcome this in tests, a workaround which assumes a single JVM setting is used. In such a setting the watermark values are stored in a static member, accessible by both the driver and the executors, bypassing the `BlockManager#putSingle(...)` `BlockManager#removeBlock(...)` APIs.

Imported from Jira [BEAM-2789](https://issues.apache.org/jira/browse/BEAM-2789). Original Jira may contain additional context.
Reported by: staslev.

Start by tracing the driver watermark update through BlockManager#removeBlock and BlockManager#putSingle, then inspect how executors request and handle a missing watermark. Review the single-JVM test workaround and the original Jira context for constraints. Done means executor requests cannot observe an unavailable intermediate watermark during an update and the relevant behavior is covered by tests.

Written by the indexing model from the issue text.