Proposal: hierarchically linked bundle Needs exports

# Hierarchically linked bundle Needs exports

## Context

The current `needs_json` action builds one composed documentation tree. A
change to any mounted `docs_bundle()` invalidates that action.

Issue #679 proposes cacheable per-bundle Needs exports and a final global
composition. That keeps bundle actions independent, but cross-bundle Need
links are deliberately unresolved in the bundle action. They are validated
only in the final composition.

This proposal explores a stricter alternative: a bundle may link only to an
explicitly declared bundle above it in the documentation hierarchy. The
target export is an input to the source bundle action, so every permitted
Need link is validated when its source bundle is built.

This is an architecture proposal for discussion, not an implementation plan.

## Goals

- Produce cacheable Needs exports at bundle granularity.
- Validate every cross-bundle Need link in the bundle that owns the link.
- Keep the Bazel graph acyclic through a directed, hierarchical dependency
model.
- Preserve stable composed `docname` paths and the existing public
`needs_json` output.
- Keep `docs_check` as the comprehensive document-level integration check.
- Avoid standalone bundle-export builds that succeed despite unresolved Need
links.

## Non-goals

- Reconstructing HTML or a complete documentation build from bundle exports.
- Allowing arbitrary, bidirectional Need links between sibling or descendant
bundles.
- Replacing Sphinx-Needs graph processing with a hand-written JSON merger.

## Proposed model

### 1. Declare an upward traceability dependency

A source-bearing `docs_bundle()` declares zero or more explicit *upward* Needs
dependencies. A dependency names an ancestor bundle in the
documentation/architecture hierarchy. The bundle provider propagates this
structural relation; `docs()` does not declare it.

Need links in the bundle's sources may target:

- Needs owned by the bundle itself; or
- Needs exported by a declared upward dependency or its upward closure.

Links to siblings, descendants, or unrelated bundles are rejected. The
declaration is a Bazel dependency, so the producer export is built before the
bundle that references it.

The declared Bazel dependency defines the allowed upward direction. It must not
be inferred from Need links, because that would make the build graph depend on
parsed documentation content.

### 2. Bind a metamodel to each bundle

Each `docs_bundle()` declares the metamodel used to build its own Needs JSON.
The declaration is optional only because omitting it selects the built-in
SCORE metamodel.

The same model must be supplied to the local source bundle implicitly created
by `docs()`, through its existing `metamodel = ...` parameter. Projects using
a custom model should wrap both macros, so the model label is configured once
in project-local Starlark rather than repeated at every call site.

A source-bearing bundle exposes its own directly buildable Needs-JSON target.
It is built with its declared metamodel and does not depend on a consuming
`docs()` target.

Each action key includes:

- the bundle's sources and direct source-link data;
- the bundle's declared metamodel and documentation toolchain;
- the bundle-relative entry document; and
- the complete transitive closure of declared upward exports.

`docs()` composes the existing bundle outputs. The build rules deliberately do
not compare metamodels; a user who composes bundles with different models owns
their compatibility.

### 3. Compose a root component and its subcomponents

A component's own documentation is the local source bundle implicitly created
by `docs()`. Its subcomponents are explicit bundles and declare that local
source bundle as their upward dependency:

```starlark
# //powertrain/BUILD.bazel
docs_bundle(
name = "engine_bundle",
source_dir = "engine/docs",
upward_bundles = [":powertrain_source_bundle"],
metamodel = ":powertrain_metamodel.yaml",
)

docs_bundle(
name = "transmission_bundle",
source_dir = "transmission/docs",
upward_bundles = [":powertrain_source_bundle"],
metamodel = ":powertrain_metamodel.yaml",
)

docs(
name = "powertrain",
source_dir = "docs",
metamodel = ":powertrain_metamodel.yaml",
bundles = [
{
"bundle": ":engine_bundle",
"mount_at": "subcomponents/engine",
},
{
"bundle": ":transmission_bundle",
"mount_at": "subcomponents/transmission",
},
],
)
```

The invocation creates `:powertrain_source_bundle` from `docs/` only, and
`:powertrain_docs_bundle` from that source bundle plus both mounted
subcomponents. `engine_bundle` and `transmission_bundle` each expose their
own directly buildable Needs-JSON target. The engine and transmission actions
import only the powertrain source export; they do not import each other.

Therefore, an engine maintainer can run:

```text
bazel build //powertrain:engine_bundle_needs_json
```

That target builds only the engine sources, its declared metamodel, and the
powertrain source export required by `upward_bundles`; it does not build the
composed documentation tree or the transmission export.

`docs()` must distinguish its own-source bundle from its composed documentation
bundle. An upward dependency may target only the own-source bundle, which has
no dependency on mounted children. The composed bundle contains the own-source
bundle plus its mounted children and is used for document-level checks. This
keeps the graph acyclic: the powertrain's composed bundle depends downward on
the engine and transmission bundles, while those exports depend upward only on
the powertrain's own-source export.

There is no self-registration: both relationships are explicit. The
powertrain declares its mounted children through `bundles`; each child declares
its parent through `upward_bundles`. A bundle export is an internal Bazel
interface: it may be consumed by declared upward/downward build relationships,
but is not a published `needs_json` artifact for external documentation
projects.

### 4. Validate links against imported upward exports

The bundle action imports the complete declared upward closure through the
supported Sphinx-Needs external-needs mechanism. Including the closure is
necessary because an imported parent may itself link to an ancestor. Their
contents are treated as validation inputs, not as part of the bundle's
published ownership.

Normal Sphinx-Needs processing then validates outgoing links, link conditions,
constraints, and duplicate IDs over the local Needs plus the allowed upward
interface. Any unresolved Need link is fatal in the bundle action.

Bundle exports must use a distinct `base_url` per imported source so duplicate
IDs cannot be silently replaced by the external-needs loader.

### 5. Publish a local export, not imported Needs

After validation, the bundle exports only Needs owned by its bundle.
Imported upward Needs are not re-exported. This preserves a small, explicit
interface and prevents transitive copies from masking ownership or duplicate
IDs.

A bundle export is an internal build artifact. Its bundle-relative `docname` is
not a published contract. Only Need IDs are stable across bundle boundaries.

### 6. Compose the final graph at the top level

The top-level `needs_json` imports all reachable local exports and runs the
normal Sphinx-Needs finalization once:

- rebase each owned Need's `docname` to its final `mount_at`;
- calculate full-graph backlinks, dead-link state, and constraints;
- run SCORE graph checks and metrics; and
- export the historical public output.

The composition must not introduce a third-party mutation patch for
Sphinx-Needs internals. It requires a supported upstream operation to
re-host a bundle Need with its final source location and local/external
semantics. The operation must be specified field by field before
implementation.

`docs_check` remains responsible for parsing, navigation, roles, document
references, rendering, and other checks requiring the mounted source tree.

## Dependency shape and cache behavior

```text
feature export
^
| validated upward link
component export
^
| validated upward link
unit export

all local exports --> lightweight top-level composition --> needs.json
```

A unit edit rebuilds its export and the top-level composition. A feature
edit also rebuilds exports that explicitly depend on that feature. Unrelated
branches remain cached.

This deliberately trades some invalidation for early, deterministic link
validation. The cost is bounded by the declared upward dependency closure,
rather than by every bundle in the documentation tree.

## Required restrictions

### Link direction

The declared `upward_bundles` Bazel dependencies define the allowed direction.
A link that cannot be classified as local or as targeting one of those
dependencies is invalid. This avoids hidden dependencies; Bazel rejects any
dependency cycle.

### `needextend`

`needextend` must be restricted to Needs owned by the same bundle (and,
if necessary, the same document). Applying it to imported upward Needs would
make an interface mutable and invalidate the ownership model.

### Dynamic functions, variants, and schema selects

Bundle-local processing must not depend on Needs outside the local-plus-
upward closure. Phase 1 must inventory dynamic functions, variants,
constraints, link conditions, and `docname`/`is_external`/`is_import`
dependent schema selects. Unsupported graph-dependent behavior must fail
clearly rather than produce an export with different semantics.

### Document references

Document references (`:doc:`, `:ref:`, includes, images) remain bundle-local
and host-to-bundle only. A bundle linter is required to make document-reference
self-containment enforceable.

### Foreign mounted bundles

The final composition must distinguish a Need that is local for rendered URL
purposes from one that remains foreign for gates and metrics. One
`is_external` flag cannot safely represent both meanings; the required
Sphinx-Needs upstream API must expose this distinction or preserve equivalent
metadata.

## Validation model

Bundle-export builds make fatal:

- duplicate IDs in the local-plus-upward closure;
- missing or disallowed Need links;
- invalid link conditions and local constraints; and
- unsupported graph-dependent features.

The top-level composition makes fatal:

- duplicate IDs across all local exports;
- invariants that require the complete graph;
- backlink and dead-link calculation failures;
- SCORE graph checks; and
- Needs-based metrics and gates.

## Suggested implementation phases

1. **Define the hierarchy contract**
- Specify the `upward_bundles` declaration and its transitive closure.
- Inventory existing cross-bundle links to assess migration feasibility.
- Decide whether multiple upward dependencies are allowed.

2. **Establish prerequisites**
- Enforce bundle-local `needextend` as fatal.
- Add a bundle linter for document-reference self-containment.
- Characterize dynamic functions, variants, constraints, link conditions,
Need parts, and schema selects.

3. **Prototype upstream integration**
- Specify and propose the supported Sphinx-Needs re-hosting API required
for final composition.
- Verify external-needs import/export behavior, ownership filtering, and
unique `base_url` handling.

4. **Build and validate bundle exports**
- Add `metamodel` to `docs_bundle()` and default it to the built-in model.
- Make every source-bearing bundle expose its own Needs-JSON target.
- Import declared upward exports and make unresolved links fatal.
- Export only locally owned Needs.
- Verify the intended cache boundaries.

5. **Compose and prove equivalence**
- Produce the final graph with normal Sphinx-Needs post-processing.
- Compare diagnostics and output with the monolithic build.
- Cover duplicate IDs, invalid directions, missing links, constraints,
Need parts, source links, metrics, and foreign mounted bundles.

6. **Migrate consumers**
- Make the composed graph the public `needs_json`.
- Move `metrics_json` and `traceability_gate` after equivalence is proven.
- Keep `docs_check` as the comprehensive integration check.

## Alternatives considered

### Fully independent exports with final-only link validation

This is the direction in issue #679. It maximizes bundle-export reuse but delays
cross-bundle link feedback until the global composition.

### Arbitrary explicit bundle dependencies

This validates links early but permits a graph that can become cyclic and
does not reflect the architectural traceability direction. Restricting
dependencies to upward edges makes the graph understandable and analyzable.

### Raw pre-post-processing exports

This may be necessary if finalized Sphinx-Needs exports cannot support the
required upstream re-hosting operation. It needs a supported serialization
boundary upstream and does not remove the need for the hierarchy contract.

## Future improvement: resolve metamodels in `docs()`

This proposal intentionally starts with a metamodel declared on every
`docs_bundle()`. It gives each bundle a directly buildable Needs-JSON target
with clear ownership and avoids the additional target-generation and
configuration-propagation machinery required for a `docs()`-owned model.

After the bundle-export model is established, we may move metamodel resolution
to `docs()`. `docs(metamodel = ...)` would then create bundle-specific
Needs-JSON actions with its resolved model, and `metamodel` would be removed
from `docs_bundle()`.

That later design would make the same bundle usable in different `docs()`
contexts with different metamodels. This is a useful side effect, not a goal
of the initial work. It also needs independently buildable generated targets
for subbundles and explicit model propagation. It is out of scope for this
proposal.

## Decision requested

Do we agree to evaluate this constrained model?

1. A bundle may link only locally or to explicitly declared upward bundles.
2. Declared upward exports are Bazel inputs and make cross-bundle links
fatal during bundle-export builds.
3. Each `docs_bundle()` declares the metamodel for its own Needs-JSON target;
projects with a custom model wrap both `docs_bundle()` and `docs()`.
4. Bundle exports include only their owned Needs; they are internal artifacts.
5. The top-level composition finalizes the complete graph and preserves
public output compatibility.
6. A supported Sphinx-Needs re-hosting API, bundle-local `needextend`, and
document-reference self-containment are prerequisites.