Smoother rust-project.json Generation

This issue is partially split off from this Zulip topic, where my understanding of @Veykril's goals is to make the extra configuration step invisible to contributors to rust-lang/rust.

For buck2, the integration that generates a rust-project.json lives in the editor via a Buck-specific VS Code extension that is called by/calls the rust-analyzer extension. This works in my situation, which is a closed-source environment where users expect teams/engineers focused on developer experience to provide tooling that just works. However, I'm not sure that's the right solution for rust-lang/rust since it'd require users to setup their editor (neovim, emacs) to run the appropriate x.py command prior to starting rust-analyzer or install an additional extension (most graphical editors, such as VS Code).

I think that moving the generate-ide-metadata command to run on the server might be the correct choice—or at the very least, an avenue worth exploring—since it's substantially easier to make it a seamless, zero-configuration experience for users. However, the following things would likely need to happen first:

• rust-analyzer.toml (https://github.com/rust-lang/rust-analyzer/issues/13529) should be implemented. A rust-analyzer.toml can then be placed at the root of the repo (or wherever appropriate!) and define what x.py needs to be run in order
  • Paired with automatic discovery of a rust-analyzer.toml, this enable setup-free IDE functionality in rust-lang/rust.
• rust-analyzer's current project loading behavior might need to be altered for different build systems:
  • For x.py generate-IDE-metadata-style commands, rust-analyzer should expose an option to tweak its current project-loading behavior to instead run that (static) command.
  • For build systems that allow querying for the owning build target/definition of a file (such as Bazel or Buck), project loading should be deferred until the user opens a file. The work I've done in https://github.com/rust-lang/rust-analyzer/pull/15863/ can repurposed/redirected to run a build-system generate-ide-metadata $opened_file in a task.
    • I've alluded to this in the past, but I think would require variable interpolation to be built into rust-analyzer so that this is less ad-hoc and users can get warnings if variables are renamed.
• I think some form of progress reporting from the generate-ide-metadata is necessary. For buck2, I've made rust-project send LSP WorkDoneProgress events to the companion VS Code extension. While that implementation is an abuse of the LSP protocol, I think a purpose-built JSON protocol for reporting progress back to rust-analyzer might be appropriate.
  • I'm not too worried about breaking change to this protocol since the blast radius of breakage is confined to a few, somewhat niche, tools.
  • Unlike how rust-project currently runs, I don't think this should be a persistent, daemonized server—it should be a one-off command.

Start by reading rust-analyzer issue 13529, the rust-analyzer project-loading behavior, and the referenced PR 15863; also inspect how rust-lang/rust invokes x.py generate-IDE-metadata. The issue names no specific files or tests, and done would require an agreed design and implementation for automatic configuration, build-system invocation, and progress reporting.