cargo doc should delete search-index.js before invoking rustdoc

Problem

Unlike the vast majority of build artifacts, search-index.js is both read and written by rustdoc, in order to append to it in a structured way.

This has the unfortunate side effect of persisting some state across cargo doc runs, as rustdoc will be able to see the output of the previous invocation.

This cannot be easily addressed within rustdoc, as it has no way to tell the difference between two invocations of rustdoc within one larger cargo doc invocation, and two different invocations of rustdoc across two different cargo doc invocations.

The weird read+write behavior of rustdoc is required because it does not have any equivalent to the "linking" step that rustc does, in which it can do one-time finalization and collection.

see https://github.com/rust-lang/rust/issues/140698 and https://github.com/rust-lang/rust/issues/133544 for some of the issues this causes

Proposed Solution

target/doc/search-index.js should be deleted by cargo doc before doing a full documentation rebuild (such as when the command-line arguments have changed, like in --document-private-items).

Notes

I believe crates.js also has a similar behavior.

Start by tracing Cargo's cargo doc full-documentation-rebuild path and how it handles target/doc/search-index.js before invoking rustdoc. Check the related crates.js behavior as well; done means stale search-index state is removed before a full rebuild without disrupting multiple rustdoc invocations within one run.