Make Table Deletion Idempotent
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
The table deletion flow was changed in #8422 where we send a message to servers and let them take care of waiting for EV convergence and then remove the segments
On the controller side, we only takes care of deleting the configs and ideal states.
However, this leads to a lot of exceptions.
For eg. when servers try to commit consuming segments it starts getting `TableNotFound` exception cause the table config was already deleted by the controller.
The new deletion flow already has the following TODO which will help solve this issue.
```java
public void deleteOfflineTable(String tableName, @Nullable String retentionPeriod) {
String offlineTableName = TableNameBuilder.OFFLINE.tableNameWithType(tableName);
LOGGER.info("Deleting table {}: Start", offlineTableName);
// Remove the table from brokerResource
HelixHelper.removeResourceFromBrokerIdealState(_helixZkManager, offlineTableName);
LOGGER.info("Deleting table {}: Removed from broker resource", offlineTableName);
// Drop the table on servers
// TODO: Make this api idempotent and blocking by waiting for externalview to converge on controllers
// instead of servers. This is because if externalview gets updated with significant delay,
// we may have the race condition for table recreation that the new table will use the old states
// (old table data manager) on the servers.
// Steps needed:
// 1. Drop the helix resource first (set idealstate as null)
// 2. Wait for the externalview to converge
// 3. Get servers for the tenant, and send delete table message to these servers
deleteTableOnServer(offlineTableName);
```
## To Reproduce:
* Start UpsertQuickStart
* Wait for some data to populate
* Delete the `upsertMeetupRsvp` table
* Check the logs for exception
Contributor guide
Research direction
Start at the deleteOfflineTable entry point and review the existing deleteTableOnServer flow and TODO. Reproduce the issue with UpsertQuickStart, then trace controller external-view convergence and server deletion behavior. Done means deletion is idempotent and blocking, avoids TableNotFound exceptions, and prevents stale states during table recreation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100