samchon / samchon/compiler-knowledge-graph

[Bulk index][Rust] Export rust-analyzer HIR snapshots beyond SCIP

Open
#72 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
6
Forks
3
PR merge metrics
No merged PRs in 30d

Description

Outcome

Add a pinned rust-analyzer graph exporter that walks one immutable HIR/IDE
revision and emits complete or explicitly bounded Rust semantic shards without
textDocument/references fan-out. Keep the already-shipped
rust-analyzer scip provider as a navigation fallback; it is not the final
provider and does not close this issue.

Current master baseline

PR #147 ships
rustScipProvider.
It requires rust-analyzer, rustc, Cargo and the scip decoder; invokes
rust-analyzer scip . --exclude-vendored-libraries; and publishes
semantic-index authority with only contains and references.

That limitation is proven in upstream source, not inferred from a sparse
fixture. Stock rust-analyzer's SCIP writer hardcodes an empty relationship list
and empty diagnostics and only marks the definition role. It cannot express
trait implementations, overrides, resolved calls, constructions, access roles,
test relationships or dispatch. Parsing formatted SCIP monikers to recover
those facts is forbidden: the rendered trait/self path is not a stable
compiler identity.

The exact PR #147 Tokio cell took 55.238 s; the strict-off rust-analyzer LSP
cell took 229.861 s. Both are cold end-to-end results. No no-op or edit
latency has been measured.

Upstream feasibility result

The final route is feasible but is not a stable crates.io sidecar.
Rust-analyzer's architecture
keeps a fully resolved incremental semantic model, applies changes through
AnalysisHost, and hands readers immutable Analysis snapshots. HIR and IDE
queries are backed by Salsa and therefore already have the invalidation behavior
the graph needs.

The API boundary is the cost:

  • cli::scip lives in the rust-analyzer binary crate;
  • workspace crates such as hir, ide, and ide_db are not a versioned
    public crates.io SDK; and
  • Cargo project loading executes build scripts and proc macros and is sensitive
    to target, cfg, features and toolchain.

Therefore choose, in order:

  1. an upstreamed versioned graphSnapshot command;
  2. a git-pinned wrapper crate built at an exact rust-analyzer commit; or
  3. a narrow maintained fork containing only the exporter and compatibility
    handshake.

A plain external parser or stock SCIP post-processor is not an acceptable final
route.

Exporter design

Add samchon/graphSnapshot or an equivalent CLI/NDJSON command in the pinned
rust-analyzer tree. For each request:

  1. capture one immutable Analysis plus the exact project-model/Cargo revision;
  2. walk crates/modules and source-owned HIR declarations once;
  3. resolve expression, type and relationship facts through HIR/IDE queries;
  4. serialize changed source/crate shards using #63's Graph Snapshot Protocol;
  5. re-check the revision and Cargo universe; and
  6. commit only if both remain unchanged.

The traversal must cover:

  • crates, modules, visibility and re-exports;
  • structs, enums, unions, traits, impls, functions, methods, associated items,
    constants, statics, fields, variants, type aliases, generics and locals;
  • resolved function/method calls, constructors, field reads/writes and type
    references;
  • direct trait implementations, supertraits and override/implementation
    relationships;
  • attributes and test functions/modules;
  • macro and proc-macro declarations/uses with expansion-to-source evidence; and
  • diagnostics from the same revision.

Do not retain rust-analyzer HIR handles across revisions. Project graph nodes
are serialized values; the next refresh reacquires semantic handles from the
new immutable Analysis.

Identity and universe

Use rust-analyzer definition identities internally, then publish a versioned
stable identity containing the Cargo package identity, crate disambiguator,
module/owner path, item kind and structural signature where overload-like
generated items require it. Do not use display-formatted monikers as the sole
key.

Explicit cases:

  • inherent and trait methods with the same name stay distinct;
  • an impl method identifies both its owning impl and implemented trait item;
  • local N-style document IDs never escape without a file/owner namespace;
  • macro-generated twins retain expansion/call-site provenance;
  • cfg-gated twins are separated by the build universe, not silently merged; and
  • generic monomorphizations do not become false source declarations.

The universe digest includes Cargo metadata, workspace/package graph, lockfile,
features (including defaults), target triple, cfg set, profile, rustc/Cargo and
rust-analyzer versions, build-script outputs, proc-macro enablement and relevant
environment/configuration. The stock SCIP artifact does not record most of
these; the provider must capture them from the project model and toolchain.

Completeness rules

Publish all 15 graph fact-family coverage rows.

  • Direct calls and inherent dispatch may be complete for the selected
    universe.
  • Trait-object and generic dispatch is partial unless rust-analyzer proves a
    closed candidate set. Candidates belong on an unresolved record, not as
    executed dispatches.
  • Code excluded by cfg/features is conditional-build, not proven absent.
  • Disabled/failed build scripts or proc macros are
    macro-or-generated/analysis-error.
  • External dependency bodies excluded by policy are external-boundary.
  • renders is unsupported unless a separately named framework enricher is
    added.

Macro-expanded facts must never masquerade as handwritten spans. Store
call-site and definition-site evidence when rust-analyzer can map both.

Incremental and performance design

Keep the rust-analyzer process resident and use its normal change path. Let
Salsa determine the affected query closure. Persist graph shards independently
from rust-analyzer's memory so process restart can validate and load an
unchanged generation.

  • unchanged revision: return the prior manifest/digest without walking every
    crate;
  • body edit: export only changed/invalidated owner/source shards;
  • public item/Cargo feature/target change: invalidate the compiler-reported
    dependent closure and report it;
  • deletion/rename: emit explicit shard deletions; and
  • crash/cancellation/revision movement: retain the last committed graph.

Cold acceptance is relative to rust-analyzer project load plus cache priming:
no declaration-proportional RPCs and no second compiler analysis. On the pinned
fixture, target resident no-op p95 below 250 ms and body-edit p95 below 2 s.

Implementation map in this repository

  1. Extend the common shard/coverage contract from #63.
  2. Add a new provider beside
    provider/rust,
    leaving rustScipProvider as fallback.
  3. Add resolver/configuration logic that pins rust-analyzer tag and commit
    and verifies rustc/Cargo compatibility before selection.
  4. Add protocol parsing and identity/coverage validation; reject unknown
    handshake versions.
  5. Add a deterministic fake-session fixture before the real upstream binary.
  6. Extend tests/experiment with the HIR provider as the preferred row and
    retain a stock-SCIP comparison row.
  7. Update the registry/docs only after the real provider passes.

Acceptance

  • No strict-HIR path invokes textDocument/references per declaration.
  • Exact fixtures cover workspaces, re-exports, features, target cfg,
    inherent/UFCS/trait calls, same-named trait methods, impls, supertraits,
    associated items, generics, closures, async, macros, proc macros, build
    output and tests.
  • Source and macro evidence is attributable to one immutable revision.
  • All 15 fact families have complete, partial, or unsupported
    coverage and unresolved reasons.
  • Unchanged, body edit, public API edit, create, delete, rename, Cargo
    metadata, feature/target, macro failure, cancel, crash and retry are
    atomic.
  • Cold time is compared with native rust-analyzer load/prime-caches; no-op
    and edit targets above are measured separately.
  • The provider reports exact rust-analyzer commit, rustc, Cargo, target,
    features and cfg universe.
  • Stock SCIP and generic LSP remain truthful fallbacks.
  • Focused tests, Rust experiment, pnpm build, pnpm test, and
    pnpm coverage pass.

Non-goals

  • inferring trait relations from SCIP strings;
  • treating potential dynamic implementations as executed targets;
  • indexing every cfg/target universe in one graph; or
  • depending on scip-rust, which is only a wrapper around the built-in
    rust-analyzer command.

Primary references

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/graph/src/provider/rust/rustScipProvider.ts and the common shard/coverage contract from #63, then read rust-analyzer's cli::scip entry point and architecture reference. Add the pinned HIR provider beside the existing Rust provider, use the deterministic fake-session fixture first, and run the focused tests plus tests/experiment; done requires the listed coverage, identity, incremental, atomicity, and performance checks to pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, typescript
Domain
compilers, devtools, testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.