search index doesn't properly regenerate when using `--document-private-items` after a normal `cargo doc` run

Hi 👋🏻,

Running the following produces a documentation where private items are unsearchable:
cargo clean && cargo doc && cargo doc --document-private-items

Whereas running cargo doc with the document-private-items flag directly produces a search index that includes private items. (cargo clean && cargo doc --document-private-items)

I cannot reproduce on a fresh minimal crate, but it is consistent when trying on https://github.com/tower-lsp-community/tower-lsp-server/.

While trying to reproduce the issue, I was curious and diffed the produced target/doc folders. Only search-index.js differs: the whole part about private items is not present in the normal then private-items run.

Meta

rustc --version --verbose:

rustc 1.82.0 (f6e511eec 2024-10-15)
binary: rustc
commit-hash: f6e511eec7342f59a25f7c0534f1dbea00d01b14
commit-date: 2024-10-15
host: x86_64-unknown-linux-gnu
release: 1.82.0
LLVM version: 19.1.1

→ also tested with rustc 1.85.0-nightly (dff3e7ccd 2024-11-26)

cc @GuillaumeGomez

Reproduce the issue with tower-lsp-server using the two cargo doc command sequences, then compare target/doc/search-index.js as described. Start from the cargo doc --document-private-items entry point and trace how the search index is generated or reused. Done means running a normal cargo doc followed by the private-items run produces an index containing private items, matching a clean private-items run.