ash-project / ash-project/spark

Track DSL module dependencies at compile-time consumption sites

Open
#290 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Elixir
Stars
204
Forks
50
Avg merge
14h 52m
Merged PRs (30d)
1

Description

- [x] I agree to follow this project's Code of Conduct
- [x] I agree to follow this project's AI Policy, or I agree that AI was not used while creating this issue.

## Is your feature request related to a problem? Please describe.

Spark currently determines module dependencies where module-valued DSL fields are declared. Unclassified module references create compile dependencies, while extension authors can opt fields out through `no_depend_modules`.

This is safe, but it over-approximates dependencies. Whether a module is required at compile time is usually known where a transformer, verifier, or persister consumes it, not where the entity or section field is declared.

As a result, extension authors need to audit all downstream consumers before deciding whether a field belongs in `no_depend_modules`. The declaration and the code that determines the actual dependency semantics are separated, so they can become inconsistent as an extension evolves.

A module may only be stored as runtime metadata:

```elixir
Transformer.persist(dsl_state, :implementation, module)
```

or its metadata may affect validation, transformation, or generated state:

```elixir
metadata = module.metadata()
```

Those uses require different dependency semantics even though the module came from the same kind of DSL field.

Structured and polymorphic fields make the declaration-site model even harder to maintain. A field can contain both runtime-only modules and modules whose metadata is consumed during compilation, as discussed in #289.

The conservative default avoids stale compile-time state, but it can also create broad incremental recompilation paths for modules that are never inspected during compilation.

## Describe the solution you'd like

I would like Spark to move dependency ownership from entity and section declarations to the transformers, verifiers, and persisters that actually consume module metadata.

The extension-authoring contract would become:

- Module-valued DSL options are dependency-free by default.
- Runtime-only consumers store or invoke the module normally.
- Consumers that only need the module to be available during the current compilation use `Code.ensure_compiled/1`.
- Consumers whose result affects compile-time validation, transformation, or generated state use a Spark helper backed by `Macro.compile_apply/4`.
- Spark obtains and supplies the caller's `Macro.Env`, so extension authors do not need to interact with compiler internals directly.
- `no_depend_modules` could eventually be deprecated because dependency-free module metadata would become the default.

Conceptually, an entity definition would no longer need to predict how every consumer will use its module fields. The removal of `no_depend_modules` is intentional:

```diff
%Spark.Dsl.Entity{
name: :action,
- no_depend_modules: [:run, :error_handler, :notifiers],
schema: [...]
}
```

Under the proposed model, module-valued DSL options would not create compile dependencies by default. A transformer, verifier, or persister would register a dependency only when it consumes module metadata during compilation.

A transformer or verifier that consumes module metadata during compilation would declare the dependency at that call site:

```elixir
metadata =
Spark.Dsl.compile_apply(
dsl_state,
module,
:metadata,
[]
)
```

A runtime-only consumer would not register a compile dependency:

```elixir
Transformer.persist(dsl_state, :implementation, module)
```

If the distinction is already represented in caller-owned DSL state, the consumer can choose the appropriate operation without inspecting the target module:

```elixir
if entity.compile_time_metadata? do
metadata =
Spark.Dsl.compile_apply(
dsl_state,
entity.implementation,
:metadata,
[]
)

use_metadata(metadata)
else
Transformer.persist(
dsl_state,
:implementation,
entity.implementation
)
end
```

The exact helper name and namespace are not important. The important change is that dependency semantics are declared where the module is consumed rather than where its field is defined.

`Macro.compile_apply/4` is available since Elixir 1.16. Raising Spark's minimum Elixir version to 1.16 does not seem unreasonable for an actively developed framework ecosystem, and it would allow Spark to provide this behavior through a public helper.

This could first be introduced as an opt-in dependency mode and become the default in a future major version after Spark-based extensions have migrated their compile-time module consumption.

## Describe alternatives you've considered

### Continue maintaining `no_depend_modules`

This is the current approach and remains the safest default. However, entity authors must understand every transformer and verifier that consumes each field, and the dependency policy can drift away from the implementation.

### Support nested dependency policies

Issue #289 proposes controlling dependencies for selected paths inside structured DSL values. This would improve precision, but dependency knowledge would still be encoded separately from the code that consumes the module.

### Domain-specific dependency callbacks

For Ash type constraints, a Type-owned callback can identify nested Types whose metadata is required during compilation. This works for that domain, but it is a specialized form of the broader consumption-site dependency model.

### Split polymorphic DSL fields

A polymorphic field can be replaced with separate fields whose dependency semantics are explicit in caller-owned DSL state. For example, the `run` and `reactor` options discussed in ash-project/ash#2885 would allow ordinary runtime implementations to remain dependency-free while Reactor modules always register a compile dependency.

This is precise for an individual DSL, but it does not provide a general extension-authoring model for Spark.

## Additional context

This proposal emerged from a sequence of concrete dependency corrections and reviews rather than from an abstract API preference:

- ash-project/ash#2885 removes compile dependencies for runtime-only DSL modules. Review of the generic action `run` field showed that a polymorphic field can mix runtime-only implementations with Reactor modules whose inputs are consumed by a compile-time verifier. That discussion also led to the possible separation of `run` and `reactor` options.
- ash-project/ash#2886 restored compile dependencies for policy check modules because their `init/1` results affect compile-time state.
- #289 explores nested dependency policies for mixed Ash type constraints. The Type-owned callback suggested there moves dependency knowledge closer to the code that understands and consumes those constraints.

Taken together, these cases suggest that much of the complexity comes from deciding dependency semantics at the declaration site rather than where module metadata is actually consumed.

I created a minimal fixture comparing `Code.ensure_compiled/1` and `Macro.compile_apply/4`:

https://github.com/jechol/elixir-conditional-compile-dependency

The result is:

| Mechanism | Compile ordering | Compile dependency |
|---|---:|---:|
| `Code.ensure_compiled/1` | Yes | No |
| `Macro.compile_apply/4` | Yes | Yes |

The fixture verifies the compiler metadata with `mix xref trace --label compile` and cross-checks the corresponding incremental compilation behavior after changing the target source. It passes with Elixir 1.16.3 / OTP 26 and Elixir 1.20.0 / OTP 29.

There is a trade-off when the dependency decision itself requires inspecting the target module. If a module is currently classified as runtime-only and therefore has no dependency, changing that same module into a compile-time implementation may not make the consumer stale.

That kind of role transition appears uncommon compared with the recurring cost of over-approximated compile dependencies. It may be reasonable to document that changing a module's compile-time role requires a clean compile, while recommending caller-owned discriminators for DSLs where automatic invalidation across role changes is important.

## Possible implementation path

If this direction is accepted, a possible implementation path would be:

1. Raise Spark's minimum Elixir version to 1.16 so it can use `Macro.compile_apply/4`.
2. Add a common Spark helper that registers a compile dependency using the DSL module's caller `Macro.Env`.
3. Introduce an opt-in mode where module-valued DSL options are dependency-free by default.
4. Dogfood the mode in Ash by migrating compile-time module consumption in its transformers, verifiers, and persisters to the new helper.
5. Document the extension-authoring contract and provide migration guidance for other Spark-based extensions.
6. Compare compile dependency graphs, incremental compilation behavior, and missed invalidation cases in real extensions.
7. If the model proves reliable, make consumption-site dependency tracking the default in a future major version.
8. Deprecate or remove `no_depend_modules` only after the default has changed and existing extensions have had a migration path.

The exact helper API, opt-in configuration, and migration timeline remain open for discussion.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Spark.Dsl.Entity dependency handling and the transformer, verifier, and persister consumption sites described in the proposal. Use the minimal fixture with `mix xref trace --label compile` to compare `Code.ensure_compiled/1` and `Macro.compile_apply/4`. Done means a defined public helper, an opt-in dependency mode, migrated compile-time consumers, and documented dependency behavior, but the exact API remains open.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
build-system, compilers, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.