thread 'rustc' panicked at compiler/rustc_metadata/src/rmeta/def_path_hash_map.rs:23:54: called `Option::unwrap()` on a `None` value

Code

Happened when running cargo nextest in https://github.com/MaterializeInc/materialize, unfortunately I couldn't reproduce it. Seems to be related to incremental compilation.

Exact command:

cargo nextest run --no-fail-fast --all-features --profile=ci --cargo-profile=ci --test-threads=96 --package=mz-adapter --package=mz-adapter-types --package=mz-dyncfg --package=mz-ore --package=mz-ore-proc --package=workspace-hack --package=mz-repr --package=mz-lowertest --package=mz-lowertest-derive --package=mz-persist-types --package=mz-proto --package=mz-build-tools --package=mz-pgtz --package=mz-ore-build --package=mz-sql-parser --package=mz-sql-lexer --package=mz-walkabout --package=mz-timely-util --package=mz-persist --package=mz-aws-util --package=mz-postgres-client --package=mz-tls-util --package=mz-storage-types --package=mz-ccsr --package=mz-cloud-resources --package=mz-server-core --package=mz-expr --package=mz-expr-derive --package=mz-expr-derive-impl --package=mz-pgrepr --package=mz-pgrepr-consts --package=mz-pgwire-common --package=mz-regexp --package=mz-sql-pretty --package=mz-expr-test-util --package=mz-repr-test-util --package=mz-interchange --package=mz-avro --package=mz-kafka-util --package=mz-ssh-util --package=mz-mysql-util --package=mz-pgcopy --package=mz-postgres-util --package=mz-rocksdb-types --package=mz-secrets --package=mz-service --package=mz-aws-secrets-controller --package=mz-build-info --package=mz-orchestrator-kubernetes --package=mz-orchestrator --package=mz-orchestrator-process --package=mz-sql-server-util --package=mz-tracing --package=mz-audit-log --package=mz-auth --package=mz-catalog --package=mz-catalog-protos --package=mz-compute-types --package=mz-controller-types --package=mz-cluster-client --package=mz-sql --package=mz-arrow-util --package=mz-cloud-provider --package=mz-dyncfgs --package=mz-metrics --package=mz-persist-client --package=mz-persist-proc --package=mz-txn-wal --package=mz-compute-client --package=mz-storage-client --package=mz-controller --package=mz-storage-controller --package=mz-durable-cache --package=mz-license-keys --package=mz-transform --package=mz-expr-parser --package=mz-segment --package=mz-timestamp-oracle --package=mz-alloc --package=mz-prof --package=mz-prof-http --package=mz-http-util --package=mz-npm --package=mz-alloc-default --package=mz-authenticator --package=mz-frontegg-auth --package=mz-dyncfg-file --package=mz-dyncfg-launchdarkly --package=mz-orchestrator-tracing --package=mz-frontegg-mock --package=mz-orchestratord --package=mz-pgwire --package=mz-pgtest --package=mz-catalog-debug --package=mz-cloud-api --package=mz-frontegg-client --package=mz-cluster --package=mz-clusterd --package=mz-compute --package=mz-storage-operators --package=mz-storage --package=mz-rocksdb --package=mz-fivetran-destination --package=mz-lsp-server --package=mz-materialized --package=mz-metabase --package=mz --package=mz-debug --package=persistcli --package=mz-s3-datagen --package=mz-sqllogictest --package=mz-testdrive --package=mz-metabase-smoketest --package=mz-test-util

Meta

rustc --version --verbose:

rustc 1.89.0 (29483883e 2025-08-04)

Error output

thread 'rustc' panicked at compiler/rustc_metadata/src/rmeta/def_path_hash_map.rs:23:54:
called `Option::unwrap()` on a `None` value
stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic
   3: core::option::unwrap_failed
   4: <rustc_query_system::dep_graph::dep_node::DepNode as rustc_middle::dep_graph::dep_node::DepNodeExt>::extract_def_id
   5: rustc_query_impl::plumbing::force_from_dep_node::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::DefIdCache<rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>>
   6: <rustc_query_impl::plumbing::query_callback<rustc_query_impl::query_impl::type_of::QueryType>::{closure#0} as core::ops::function::FnOnce<(rustc_middle::ty::context::TyCtxt, rustc_query_system::dep_graph::dep_node::DepNode, rustc_query_system::dep_graph::serialized::SerializedDepNodeIndex)>>::call_once
   7: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
   8: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
   9: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
  10: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
  11: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
  12: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
  13: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
  14: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
  15: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
  16: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
  17: <rustc_query_system::dep_graph::graph::DepGraphData<rustc_middle::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl::plumbing::QueryCtxt>
  18: rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::DefaultCache<rustc_type_ir::canonical::CanonicalQueryInput<rustc_middle::ty::context::TyCtxt, rustc_middle::ty::ParamEnvAnd<rustc_middle::ty::predicate::Predicate>>, rustc_middle::query::erase::Erased<[u8; 2]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, true>
  19: <rustc_trait_selection::traits::fulfill::FulfillProcessor as rustc_data_structures::obligation_forest::ObligationProcessor>::process_obligation
  20: <rustc_data_structures::obligation_forest::ObligationForest<rustc_trait_selection::traits::fulfill::PendingPredicateObligation>>::process_obligations::<rustc_trait_selection::traits::fulfill::FulfillProcessor>
  21: rustc_traits::codegen::codegen_select_candidate
      [... omitted 3 frames ...]
  22: rustc_monomorphize::collector::find_tails_for_unsizing
  23: rustc_monomorphize::collector::items_of_instance
      [... omitted 1 frame ...]
  24: rustc_monomorphize::collector::collect_items_rec
  25: rustc_monomorphize::collector::collect_items_rec
  26: rustc_monomorphize::collector::collect_items_rec
  27: rustc_monomorphize::collector::collect_items_rec
  28: rustc_monomorphize::collector::collect_items_rec
  29: rustc_monomorphize::collector::collect_items_rec
  30: rustc_monomorphize::collector::collect_items_rec
  31: rustc_monomorphize::collector::collect_items_rec
  32: rustc_monomorphize::collector::collect_items_rec
  33: rustc_monomorphize::collector::collect_items_rec
  34: rustc_monomorphize::collector::collect_items_rec
  35: rustc_monomorphize::collector::collect_items_rec
  36: rustc_monomorphize::collector::collect_items_rec
  37: rustc_monomorphize::collector::collect_items_rec
  38: rustc_monomorphize::collector::collect_items_rec
  39: rustc_monomorphize::collector::collect_items_rec
  40: rustc_monomorphize::collector::collect_items_rec
  41: rustc_monomorphize::collector::collect_items_rec
  42: rustc_monomorphize::collector::collect_items_rec
  43: rustc_monomorphize::collector::collect_items_rec
  44: rustc_monomorphize::collector::collect_items_rec
  45: rustc_monomorphize::collector::collect_items_rec
  46: rustc_monomorphize::collector::collect_items_rec
  47: rustc_monomorphize::collector::collect_items_rec
  48: rustc_monomorphize::collector::collect_items_rec
  49: rustc_monomorphize::collector::collect_items_rec
  50: rustc_monomorphize::collector::collect_items_rec
  51: rustc_monomorphize::collector::collect_items_rec
  52: rustc_monomorphize::collector::collect_items_rec
  53: rustc_monomorphize::collector::collect_items_rec
  54: rustc_monomorphize::collector::collect_items_rec
  55: rustc_monomorphize::collector::collect_items_rec
  56: rustc_monomorphize::collector::collect_items_rec
  57: rustc_monomorphize::collector::collect_items_rec
  58: rustc_monomorphize::collector::collect_crate_mono_items::{closure#1}::{closure#0}
  59: rustc_monomorphize::partitioning::collect_and_partition_mono_items
      [... omitted 6 frames ...]
  60: <rustc_metadata::rmeta::encoder::EncodeContext>::encode_crate_root
  61: rustc_metadata::rmeta::encoder::encode_metadata
  62: rustc_metadata::fs::encode_and_write_metadata
  63: <rustc_interface::queries::Linker>::codegen_and_build_linker
  64: rustc_interface::passes::create_and_enter_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}
  65: rustc_interface::interface::run_compiler::<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
note: rustc 1.89.0 (29483883e 2025-08-04) running on x86_64-unknown-linux-gnu
note: compiler flags: --crate-type lib -C opt-level=3 -C lto=off -C embed-bitcode=no -C debuginfo=line-tables-only -C debug-assertions=on -C overflow-checks=off -C incremental=[REDACTED] -C link-arg=-Wl,--compress-debug-sections=zlib -C link-arg=-Wl,-O3 -C link-arg=-fuse-ld=lld -C symbol-mangling-version=v0 -C target-cpu=x86-64-v3 -C target-feature=+aes,+pclmulqdq
note: some of the compiler flags provided by cargo are hidden
query stack during panic:
#0 [evaluate_obligation] evaluating trait selection obligation `alloc::sync::Arc<std::sync::poison::mutex::Mutex<mz_cluster::client::TimelyContainer<mz_compute::server::Config>>>: core::marker::Send`
#1 [codegen_select_candidate] computing candidate for `<core::pin::Pin<alloc::boxed::Box<{async block@<mz_cluster::client::ClusterClient<mz_compute::server::Config> as mz_service::client::GenericClient<<mz_compute::server::Config as mz_cluster::client::ClusterSpec>::Command, <mz_compute::server::Config as mz_cluster::client::ClusterSpec>::Response>>::send<'_, '_>::{closure#0}}>> as core::ops::unsize::CoerceUnsized<core::pin::Pin<alloc::boxed::Box<dyn core::future::future::Future<Output = core::result::Result<(), anyhow::Error>> + core::marker::Send>>>>`
#2 [items_of_instance] collecting items used by `<mz_cluster::client::ClusterClient<mz_compute::server::Config> as mz_service::client::GenericClient<<mz_compute::server::Config as mz_cluster::client::ClusterSpec>::Command, <mz_compute::server::Config as mz_cluster::client::ClusterSpec>::Response>>::send::<'_, '_>`
#3 [collect_and_partition_mono_items] collect_and_partition_mono_items
#4 [exported_symbols] collecting exported symbols for crate `0`
end of query stack
there was a panic while trying to force a dep node
try_mark_green dep node stack:
#0 TraitSelect(9e3ce60b982779b3-c2de4104fc0f9e04)
#1 TraitSelect(d7c4316d29fa8b8f-95c26c463a653265)
#2 TraitSelect(bbb0a20f3678938e-4572724ef43b527a)
#3 TraitSelect(2771842af4df87c9-d1d6190a70fe7adc)
#4 TraitSelect(d23ead82a2c89877-409bed86821dbbd6)
#5 TraitSelect(818d873be2da2589-25c6689504a84a5a)
#6 TraitSelect(458d8189452efc89-5867d03603003152)
#7 TraitSelect(aafc1bf09754b6eb-661c26c157368ca2)
#8 TraitSelect(26aa21aa8aee2230-a28d0c9a12387fb8)
#9 TraitSelect(933026ab011bc129-fd3e703ae49544f)
#10 evaluate_obligation(a01fd1142099025a-9d475cab2a9aa7fe)
end of try_mark_green dep node stack
there was a panic while trying to force a dep node
try_mark_green dep node stack:
#0 exported_symbols(mz_clusterd[fbe2])
end of try_mark_green dep node stack

Maybe related to https://github.com/rust-lang/rust/issues/147060

Start with compiler/rustc_metadata/src/rmeta/def_path_hash_map.rs and the reported rustc query stack, especially the incremental-compilation and dep-node context. Try the supplied cargo nextest command with rustc 1.89.0 and the listed compiler flags; done means identifying a reproducible cause and preventing this compiler panic.