apache / apache/fluss

[coordinator] Coordinator server may fail to create partition by auto partition when restart

Open
#1,619 2 comments 0 reactions 1 assignee Claimed by @zcoo View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

main (development)

### Please describe the bug 🐞

It seems a bug only happens when Coordinator server restarts.

I reproduce it by following steps:
**step 1**: Create a partition table with auto create partition on.
**step 2**: Restart Coordinator server.
**step 3**: Coordinator server starts up in the following code:

```

1. public void startup() {
2. coordinatorContext.setCoordinatorServerInfo(getCoordinatorServerInfo());
3. // start watchers first so that we won't miss node in zk;
4. tabletServerChangeWatcher.start();
5. tableChangeWatcher.start();
6. LOG.info("Initializing coordinator context.");
7. try {
8. initCoordinatorContext();
9. } catch (Exception e) {
10. throw new FlussRuntimeException("Fail to initialize coordinator context.", e);
11. }
12.
13. // We need to send UpdateMetadataRequest after the coordinator context is initialized and
14. // before the state machines in tableManager are started. This is because tablet servers
15. // need to receive the list of live tablet servers from UpdateMetadataRequest before they
16. // can process the LeaderRequests that are generated by replicaStateMachine.startup() and
17. // partitionStateMachine.startup().
18. // update coordinator metadata cache when CoordinatorServer start.
19. HashSet tabletServerInfoList =
20. new HashSet<>(coordinatorContext.getLiveTabletServers().values());
21. serverMetadataCache.updateMetadata(
22. coordinatorContext.getCoordinatorServerInfo(), tabletServerInfoList);
23. updateTabletServerMetadataCacheWhenStartup(tabletServerInfoList);
24.
25. // start table manager
26. tableManager.startup();
27.
28. // start the event manager which will then process the event
29. coordinatorEventManager.start();
30. }

```

**step 4**: AutoPartitionManager starts and trigger to auto create new partition exactly in `initCoordinatorContext();`. However partition creation needs to get alive servers for assignment from metadataCache.
**step 5**: serverMetadataCache.updateMetadata() (line 21) is later than initContext() (line 8), so step 4 get empty alive server list and fail to auto create partition with exception info:
`com.alibaba.fluss.exception.InvalidReplicationFactorException: Replication factor: 3 larger than available tablet servers: 0."
`
### Solution

Update server metadata before `initCoordinatorContext()`

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.