[BUG] Data sent to the store immediately after the store startup/initialization can be lost

Nobody has claimed this yet.

### Willingness to contribute

Yes. I can contribute a fix for this bug independently.

### Venice version

latest dev as of May 1st 2023

### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 20.0)**: macOS
- **JDK version**:
```
$ java -version
openjdk version "17.0.3" 2022-04-19
OpenJDK Runtime Environment Temurin-17.0.3+7 (build 17.0.3+7)
OpenJDK 64-Bit Server VM Temurin-17.0.3+7 (build 17.0.3+7, mixed mode, sharing)
```

### Describe the problem

https://venicedb.slack.com/archives/C03SLQWRSLF/p1682719789550259

I am using VeniceSystemProducer to send the data like
```java
producer.put(key, value).whenComplete((___, error) -> {
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("Processed key: {}, value: {}", key, value);
}

if (error != null) {
LOGGER.error("Error handling record with key {}", key, error);
record.fail();
} else {
if (safePutToQueue(record)) {
maybeSubmitFlush();
}
}
});
```

What I found out is that unless I wait (like Thread.sleep(30000)) after store init/empty push I get no errors on write but I also don't get the data. After I added the sleep the data sent to the store shows up in get requests.
The store is in ONLINE state, I tried waiting like this and it passes the check immediately:
```java
Awaitility.await().atMost(90, TimeUnit.SECONDS).untilAsserted(() -> {
ExecResult res = execByService(
"venice-client",
"bash",
"-c",
"java -jar " + jar + " --describe-store --url " + veniceControllerUrl
+ " --cluster " + clusterName
+ " --store " + storeName);

assertTrue(res.getStdout().contains("\"status\" : \"ONLINE\""));
});
```

Testcontainers starts venice components then the test runs "--new-store"/"--update-store "/"--empty-push" using admin tool. This should be enough to start submitting the data.

Expected:
producer.put() to actually put the data into the store or fail.
Ideally a way to check the store for readiness.

### Tracking information

_No response_

### Code to reproduce bug

Will be available in PulsarVeniceSinkTest once the PR is ready for submission

### What component(s) does this bug affect?

- [ ] `Controller`: This is the control-plane for Venice. Used to create/update/query stores and their metadata.
- [ ] `Router`: This is the stateless query-routing layer for serving read requests.
- [ ] `Server`: This is the component that persists all the store data.
- [ ] `VenicePushJob`: This is the component that pushes derived data from Hadoop to Venice backend.
- [ ] `Thin Client`: This is a stateless client users use to query Venice Router for reading store data.
- [ ] `Fast Client`: This is a stateful client users use to query Venice Server for reading store data.
- [ ] `Da Vinci Client`: This is an embedded, stateful client that materializes store data locally.
- [ ] `Alpini`: This is the framework that fast-client and routers use to route requests to the storage nodes that have the data.
- [X] `Samza`: This is the library users use to make nearline updates to store data.
- [ ] `Admin Tool`: This is the stand-alone client used for ad-hoc operations on Venice.
- [ ] `Scripts`: These are the various ops scripts in the repo.

Start with the VeniceSystemProducer flow and the planned PulsarVeniceSinkTest reproduction, then examine how store initialization and the ONLINE check relate to Samza writes. Reproduce the startup sequence with --new-store, --update-store, and --empty-push. Done means data sent immediately after initialization is stored or producer.put() reports an error, with a readiness check if needed.

Written by the indexing model from the issue text.