Atomic dataset deletion

Motivation

Right now, to delete a dataset we delete all the files. This causes concurrent readers and writers to hit random "not found" errors. Even worse, if deletion fails halfway, this can leave the dataset in a corrupt state with missing files.

We also want to make it safe to immediately re-create a dataset at the same location, which is a common pattern many users expect. Currently it's safest to do write_dataset(..., mode="overwrite"). But if we could make dropping the table and recreating it just as safe, that would remove a major footgun.

Solution

1. Create a new manifest state for a soft deleted table: zero fragments and zero fields in the schema.
   • Any concurrent reader / writer should report the dataset as being dropped when in this state.
2. Create a new transaction type to delete a dataset. It should be compatible on top of any other transaction, but all other transactions except possibly overwrite should fail.
3. Add a method (maybe part of cleanup) that does hard deletion of the file, but in a specific order: Old manifests should go first, then all data files, and finally the latest manifest. This order determines no concurrent readers should load a manifest and then try to read a file that isn't there.
4. Allow write_dataset in create or overwrite mode to write to the dropped dataset.

Start at the dataset deletion and write_dataset entry points, then trace manifest states, transaction handling, and cleanup. Review how readers and writers interpret manifests and how files are removed. Done means deletion is atomic for concurrent access, failed cleanup cannot corrupt the dataset, recreation at the same location is safe, and the specified hard-deletion order is enforced.