[Feature] Optimize macro namespace construction to reduce parse time
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Is this your first time submitting a feature request?
- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
Requesting more efficient resolution of macros during dbt parse time to reduce parse times.
In our (rather large) dbt project, inefficient macro namespace construction results in an estimated 73.07s of parse time overhead (~30% of parse time).
**Context**: During parse, dbt builds a `MacroNamespace` for every node so that `{{ ... }}` rendering can look up macros (`dbt.run_query`, `my_pkg.my_macro`, etc.). Today the namespace is rebuilt from scratch per node. Every macro in every package is wrapped in a fresh `MacroGenerator`, the package-level dict is constructed, and the global/dbt namespace is composed. On a 19k-path project this is roughly N×M work where N is "nodes parsed" and M is "macros visible to that node." The same work is repeated thousands of times because the macro graph itself is invariant for the duration of a parse.
### Describe alternatives you've considered
1. Switch to dbt-fusion: we'll be doing this locally but it's not robust enough for prod usage yet.
2. Optimize something else: we're also looking into other options, but this is the lowest hanging fruit based on our profiling analysis.
### Who will this benefit?
All users of dbt-core. Benefit will be larger for those with larger projects -- specifically those using many macros or with many models. That said, I tried this out and saw perf improvements even for a small project with a few hundred models when many macros get pulled in by several dbt packages.
### Are you interested in contributing this feature?
Yes, just need review of the idea and code contributions.
### Anything else?
**Proposed implementation:** lazy-load macro namespaces by restructuring the namespace into two tiers:
1. a **per-`Manifest` `NamespaceTemplate`** that is computed once and captures the package layout + which macros are visible from where, and
2. a **per-node `LazyMacroNamespace`** that defers `MacroGenerator` construction until a macro is actually accessed during rendering.
Most macros are never touched by most nodes. So the lazy path makes the per-node cost proportional to *used* macros rather than *visible* macros.
**Details:**
1. **`Manifest.get_namespace_template(root_package, search_package, internal_packages)`** computes a `NamespaceTemplate` that holds, per package, the dict of `(macro_name → unique_id)` visible from that root/search pair. This is pure metadata — no `MacroGenerator` instances, no node binding. It's memoized on the `Manifest`, so it's built at most once per (root, search, internals) tuple per Manifest instance.
2. **`LazyMacroNamespace(manifest, template, node, ...)`** is a `MacroNamespace` subclass that wraps the template and constructs `MacroGenerator` instances on first access via a `MacroDictProxy`. This is a dict subclass whose magic methods materialize the generator on demand and cache it.
3. **Unit-test override compatibility**: `generate_runtime_unit_test_context` mutates the namespace to inject test overrides, but the lazy package view is read-only. So the override path materializes affected packages to a plain dict before mutating.
[Example implementation is here.](https://github.com/macks22/dbt-core/compare/macks22:aee6b3c...macks22:616a1de)
Contributor guide
Assessment
This issue has not been assessed yet.