eclipse-score / eclipse-score/docs-as-code

Design cacheable per-bundle Needs exports and global graph composition

Open
#679 2 comments 0 reactions 3 assignees Claimed by @MaximilianSoerenPollak View on GitHub
Dominant language
Python
Stars
10
Forks
32
Avg merge
23h 52m
Merged PRs (30d)
52

Description

## Context

The public `needs_json` target currently runs one Sphinx build over the complete
mounted documentation tree. A change in one `docs_bundle()` therefore
invalidates a large global action and makes unchanged bundles rebuild their
Needs data.

We should agree on a long-term architecture before extending the current bundle
implementation further. The intended direction is to make a bundle the unit of
ownership and caching for the documentation, implementation links, and
eventually test metadata of a component, feature, or unit.

This issue proposes an architecture for discussion. It does not prescribe the
exact implementation yet.

## Goals

- Produce cacheable Needs data per source-bearing `docs_bundle()`.
- Do not invalidate unrelated bundle exports when one bundle changes.
- Keep final `docname` paths compatible with the mounted documentation tree.
- Preserve global Sphinx-Needs validation instead of replacing it with a
hand-written JSON merger.
- Keep the existing comprehensive `docs_check` for checks that require the
complete source tree or doctrees.
- Avoid mandatory Needs dependencies between bundles.

## Why bundles should not depend on every referenced bundle

Needs links naturally cross architectural levels and may run in both
directions. Making these references Bazel dependencies would likely create
cycles between units, components, and features. It would also mean that a
small documentation edit at a higher level invalidates lower-level bundle
exports.

The proposed architecture is therefore a fan-in model:

```text
bundle A sources -> bundle A needs.json --\
bundle B sources -> bundle B needs.json ----> lightweight global Needs build
bundle C sources -> bundle C needs.json --/
```

Explicit bundle dependencies may remain an option for independently published
subsets, but should not be required by the base model.

## Proposed architecture

### 1. Standalone bundle exports

Every source-bearing `docs_bundle()` produces a standalone `needs.json` using
the Sphinx Needs builder.

The action consumes only:

- the bundle's own documentation sources;
- its `entry_doc`;
- the fixed docs-as-code extension stack and built-in metamodel;
- its direct source-link data;
- bundle-local test metadata when available.

It does not consume parent or sibling sources, final `mount_at` placement, or
the complete composed bundle.

The export uses a generated minimal Sphinx configuration. Need types, fields,
and link definitions are already part of the docs-as-code toolchain and do not
need to become parameters of every bundle.

### 2. Lightweight global Sphinx-Needs build

The top-level `needs_json` target uses a synthetic Sphinx project without the
mounted RST sources. It imports all reachable bundle exports through the
existing `needs_external_needs` mechanism.

Sphinx-Needs initially represents such imports as external Needs and removes
local source information. Before normal Needs post-processing, a docs-as-code
handler would relocalize only the mounted bundle imports:

- restore intrinsic fields from the original bundle export;
- rebase `docname` using the bundle's final `mount_at`;
- restore source location data;
- set `is_external` to `False`;
- remove temporary external URL metadata;
- leave backlinks, dead-link flags, and constraint results unset.

Projects imported through `docs(data=[...])` remain truly external and are not
relocalized.

### 3. Let Sphinx-Needs finalize the graph

After relocalization, normal Sphinx-Needs post-processing runs over the complete
graph. In the currently pinned Sphinx-Needs version this includes:

- link and link-condition resolution;
- resetting and regenerating backlinks;
- dead-link calculation;
- constraint processing;
- final Needs sealing and export.

This is preferable to directly merging finalized JSON and manually reproducing
Sphinx-Needs semantics.

### 4. Keep the full documentation check

The lightweight global Needs build becomes authoritative for the Needs graph,
but cannot replace checks that require mounted source documents.

`docs_check` remains responsible for:

- RST/MyST parsing;
- roles and document references;
- toctrees and navigation;
- extension behavior requiring doctrees;
- rendering and assets.

## Expected cache behavior

For a source change in bundle A:

- bundle A export rebuilds;
- bundle B and C exports remain cached;
- the lightweight global Needs build rebuilds;
- the full `docs_check` rebuilds only when requested.

For a placement-only change, all bundle exports remain cached and only the
global composition rebuilds.

## Semantic assumptions and questions

### `needextend`

Our metamodel restricts `needextend` to the current document. It can therefore
be applied completely inside the standalone bundle build and cannot mutate a
Need owned by another bundle.

### Dynamic functions and variants

Bundle exports contain values after local Sphinx-Needs processing. A dynamic
function that queries Needs outside the bundle may therefore produce a
different result from the monolithic build.

Before adopting the architecture, we need to inventory current usage and
either:

- prove that used functions depend only on local Need/document/configuration
data;
- reject graph-dependent functions for bundle exports; or
- fall back to exporting pre-postprocessing fragments if this is a real
requirement.

### External import round trip

We need to establish exactly which fields are removed by the external loader
and which must be restored. Globally derived fields must not be restored,
because Sphinx-Needs should recalculate them.

The implementation must also preserve link conditions and Need parts across
the export/import round trip.

### Custom metamodels

The normal Needs dialect is built into docs-as-code. We still need to decide
how `docs(metamodel=...)` behaves:

- validation-only customization can remain a global input;
- customization that changes directives, fields, or links may require a
compatible bundle build or a monolithic fallback.

## Validation model

Bundle builds have only a partial graph. They may suppress narrowly classified
warnings for unresolved cross-bundle links, while parse errors and local checks
remain fatal.

The global Needs build must make Needs-level warnings fatal and run:

- duplicate-ID validation;
- outgoing-link and link-condition validation;
- backlink and dead-link calculation;
- constraints;
- SCORE graph checks;
- Needs-based metrics.

The full `docs_check` remains the final integration check for all
document-level behavior.

## Suggested implementation phases

1. **Characterize semantics**
- Inventory dynamic functions, variants, constraints, link conditions, and
Need parts.
- Add equivalence fixtures for the features we use.
- Decide custom-metamodel behavior.

2. **Add standalone exports**
- Produce one cacheable `needs.json` per source bundle.
- Feed direct source links into the bundle action.
- Propagate exports and source runtime paths through `DocsBundleInfo`.
- Verify that unrelated bundle actions stay cached.

3. **Add global composition**
- Import bundle exports as External Needs.
- Relocalize mounted bundle Needs before post-processing.
- Run the normal Sphinx-Needs finalization and SCORE graph checks.
- Preserve the historical public output path.

4. **Prove equivalence**
- Compare the new output and diagnostics with the monolithic build.
- Cover duplicate IDs, missing links, constraints, Need parts, true external
Needs, source links, and metrics.

5. **Switch consumers**
- Make the composed graph the public `needs_json`.
- Move `metrics_json` and `traceability_gate` only after graph equivalence is
established.
- Keep `docs_check` as the comprehensive integration check.

## Alternatives considered

### Direct JSON merge

Simple, but derived graph state would have to be manually removed or
recalculated. This risks weakening validation and duplicating Sphinx-Needs
semantics.

### Raw pre-postprocessing fragments

Semantically stronger for graph-dependent dynamic functions, but requires a
new stable serialization/import boundary in Sphinx-Needs. This remains the
fallback if finalized bundle exports cannot be composed correctly.

### Explicit dependencies between bundles

Provides earlier link checking, but introduces architectural coupling, cache
invalidation, and likely dependency cycles. Keep as an optional model, not the
default.

## Decision requested

Do we agree on the following direction?

1. `docs_bundle()` is the cache unit for local Needs extraction.
2. Bundle exports do not depend on other mounted bundles.
3. The global graph is produced by importing, relocalizing, and normally
post-processing all bundle exports in a lightweight Sphinx build.
4. `docs_check` remains the comprehensive document-level integration check.
5. We prototype and verify semantic equivalence before switching public
consumers.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.