Remove PDS metadata before dropping tables in manage-partitions delete command

Remove PDS metadata before dropping tables in manage-partitions delete command

In the manage-partitions delete command we wait for the tables to drop before removing the entries in the catalog. If the tables take a long time to drop and we get an rpc timeout the command fails to remove the metadata entries. The timeout just means the process is still running and the wait period has passed. We should delete the metadata first and tell the user that the RPC has timed out and to check HBase to make sure the tables drop successful.

If there is another issue with the drop the user can re-adopt the tables into the PDS in order to run the delete command again or manually remove them from HBase.

Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2645

Key: GEOMESA-2645
Type: Improvement
Priority: Major
Status: To Do
Resolution: Unresolved
Assignee: Austin Heyne
Reporter: Austin Heyne
Created: Wed, 5 Jun 2019 15:46:24 -0400
Updated: Wed, 15 Apr 2020 11:20:06 -0400

Comments:

557058:48912df4-0004-4f90-9915-db26684ec006 (Tue, 14 Apr 2020 15:56:50 -0400):

Emilio Lahr-Vivaz Austin Heyne how important is this? And what’s the fix look like?

557058:da7f6dfa-f0a1-468b-aec5-2cb61273feb5 (Tue, 14 Apr 2020 16:25:25 -0400):

Looking at the code, this has been partially fixed. Metadata entries are deleted first, then the tables are dropped. However, we don't have any specific handling or warnings around errors/timeouts deleting tables.

557058:3f10fbb6-8894-45e8-828a-0f6aa4281f1d (Wed, 15 Apr 2020 11:20:06 -0400):

Deleting tables on S3 can take hours/days so don’t know how much it’s worth handling that better. As long as the meta is clean you can delete the tables manually. We should definitely tell the user that we timed out waiting or there may be unexpected behavior if that partition gets re-created and some index tables still exist.

Start at the manage-partitions delete command and inspect the existing metadata-first flow and table-drop wait handling. Add user-facing handling for RPC timeouts or table-drop errors, explaining that the user should check HBase and that metadata has already been removed; verify the command behavior for slow or unsuccessful drops.