Proposal: Extend `exists/2` to traverse embedded array attributes
- Dominant language
- Elixir
- Stars
- 2.5k
- Forks
- 426
- Avg merge
- 23h 26m
- Merged PRs (30d)
- 46
Description
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
### AI Policy
- [x] I agree to follow this project's AI Policy, or I agree that AI was not used while creating this issue.
### Is your feature request related to a problem? Please describe.
Ash expressions today let you ask "does this collection contain an element satisfying P?" over relationships via `exists/2`, but there is no expression-level equivalent for an embedded array attribute (`{:array, EmbeddedResource}`).
| Shape | Expression |
|---|---|
| `belongs_to` / `has_one` | `rel.field == x` |
| `has_many` / `many_to_many` | `rel.field == x` (auto-any) or `exists(rel, ...)` |
| Single embedded attribute | `attr[:field]` |
| **Embedded array attribute (`{:array, EmbeddedResource}`)** | **— no expression — must use `fragment`** |
To ask "any line item's name contains `word`" over a resource shaped like `Estimate.options :: {:array, Option}` / `Option.line_items :: {:array, LineItem}`, the only option today is to drop to raw SQL:
```elixir
expr(fragment(
"""
EXISTS (
SELECT 1
FROM jsonb_array_elements(?) AS opt,
jsonb_array_elements(opt->'line_items') AS li
WHERE li->>'name' ILIKE ?
)
""",
options,
^"%#{word}%"
))
```
This:
- bypasses Ash's type system and predicate composition,
- leaks data-layer specifics (`jsonb_array_elements`, `->>`) into application code,
- requires manual parameter handling,
- is verbose enough that teams denormalize prematurely or skip the feature.
From the user's perspective, "this collection contains an element satisfying P" is the same question regardless of whether the collection is modeled as a `has_many` or an embedded array. The expression surface should reflect that.
### Describe the solution you'd like
Generalize `exists/2` so each path segment is resolved against the resource schema and may be **either**:
- a relationship segment (existing behavior — join / `EXISTS` translation), or
- an attribute segment of type `{:array, EmbeddedResource}` where `EmbeddedResource` is `use Ash.Resource, data_layer: :embedded` (new — unnest the array, evaluate the predicate per element).
The user-facing API is unchanged; dispatch is per-segment and internal.
```elixir
# existing — relationship
exists(invoices, state == :paid)
# new — embedded array
exists(options, total_amt > 100)
# new — nested embedded arrays
exists(options.line_items, contains(name, ^word))
# new — mixed path (relationship → embedded array)
exists(invoices.options, contains(name, ^word))
```
Inside the predicate, fields resolve against the innermost embedded resource's attributes; further nesting into single embeds keeps the existing bracket syntax (`address[:city]`). `^arg/1`, nested `exists`, and predicate composition behave identically to the relationship case.
### Where the change lives in core
- `expand_through_path/3` (`lib/ash/filter/filter.ex:5206-5229`) is the single dispatch point that today raises `NoSuchRelationship` for any non-relationship segment. The fallback to recognize `{:array, EmbeddedResource}` attributes (and carry the inner resource for the next segment) belongs here.
- `Ash.Query.Exists` (`lib/ash/query/function/exists.ex`) and `do_hydrate_refs`'s `Exists` branch (`lib/ash/filter/filter.ex:4615-4654`) need to carry per-segment kind so downstream consumers can dispatch correctly. Happy to follow maintainers' preference on whether to widen the existing struct or introduce a sibling concept.
- `Ash.Filter.Runtime` and `Ash.Filter.match?/2` need matching support — see point 6 below.
### Design decisions to pin down before implementation
Listed with my suggested default for each. Happy to revise based on maintainer input.
1. **Where does an unsupported-segment error surface?** At query-build time, via a data-layer capability check (e.g., `Ash.DataLayer.can?(data_layer, :exists_over_embedded_array)`), so the failure surfaces at the same point other unsupported expressions surface today — not silently at expression construction, not at execute time.
2. **`parent/1` semantics inside the predicate.** Refers to the immediately enclosing scope, matching how nested `exists` over relationships works today. In `exists(options.line_items, parent(...))`, `parent` refers to an element of `options`. Outer scopes are reachable by nesting `parent` calls.
3. **`at_path` interaction.** `at_path` segments may include embedded-array attributes under the same rules as the main path. No parsing change to `at_path` itself.
4. **Auto-any for embedded arrays.** **No.** `rel.field == x` auto-anys over a `has_many` today; permitting the same shorthand for embedded arrays would collide with single-embed access semantics and make `options.total_amt > 100` ambiguous depending on whether `options` is `Option` or `{:array, Option}`. Require explicit `exists/2` for embedded arrays.
5. **What counts as an "embedded array".** `{:array, X}` where `X` is a module declared with `use Ash.Resource, data_layer: :embedded`. Primitive arrays (`{:array, :string}`, `{:array, :integer}`, ...) and arrays of custom `Ash.Type` modules are explicitly out of scope — there is no per-element resource schema for fields to resolve against.
6. **In-memory runner is in scope, not optional.** `Ash.Filter.Runtime` and `Ash.Filter.match?/2` must handle the new segment kind from day one. Policy evaluation and several data layer code paths fall back to in-memory filtering; treating runtime support as "if/when wanted" would silently break those paths.
7. **Sort and aggregates over embedded arrays.** Out of scope. They require giving the embedded array a query-time identity (load lifecycle, calc loading, …) which is a much larger redesign. Filter-only has clean semantics; sort/aggregate do not without additional groundwork.
### Illustrative SQL translation (AshPostgres)
`exists(options.line_items, contains(name, ^word))`:
```sql
EXISTS (
SELECT 1
FROM jsonb_array_elements("t0"."options") AS opt(elem),
jsonb_array_elements(opt.elem -> 'line_items') AS li(elem)
WHERE li.elem ->> 'name' ILIKE $1
)
```
Mixed path `exists(invoices.options, total_amt > 100)`:
```sql
EXISTS (
SELECT 1
FROM "invoice" AS inv
CROSS JOIN LATERAL jsonb_array_elements(inv."options") AS opt(elem)
WHERE inv."" = "t0"."id"
AND (opt.elem ->> 'total_amt')::numeric > 100
)
```
Per-attribute casts (text via `->>`, numeric via `::numeric`, etc.) reuse the existing AshPostgres single-embed access infrastructure.
### Describe alternatives you've considered
1. Add a separate `any/2` macro for embedded arrays; keep `exists/2` for relationships.**
Rejected. The two operations are the same existential predicate from the user's perspective; splitting them by collection kind forces users to track whether a field is modeled as `has_many` or as an embedded array, and to rewrite expressions whenever the modeling changes (e.g., adding snapshotting later). Mixed paths (relationship → embedded array) compose naturally with one macro but would require awkward interleaving with two.
**2. Bracket-projection syntax (`options[:*][:line_items][:*][:name]`).**
Rejected as the primary surface. It conflates two distinct meanings — list projection vs. existential predicate target — and its semantics under multiple conditions in one expression are ambiguous (does `options[:*][:line_items][:*][:name] == X and ...price > Y` require the *same* line item to satisfy both, or any element to satisfy each independently?). A projection form for calculations returning lists could still be introduced separately; orthogonal to this proposal.
**3. Keep status quo and continue using `fragment(...)`.**
Rejected as a long-term answer for the reasons in the problem section: it leaks data-layer details, bypasses type checking and predicate composition, and has no story for non-Postgres data layers.
### Additional context
Backwards compatibility.** Pure addition. `exists(options, ...)` where `options` is an embedded array attribute currently raises `NoSuchRelationship` at expression construction; under this proposal it gains a defined meaning. No existing `exists/2` call changes behavior.
**Authorization.** Embedded resources are not separately authorized today; the parent resource's policies gate visibility of the whole row, embedded array included. Allowing filter predicates over embedded array contents does not introduce a new leak vector — if the parent row is visible, its embedded values were already accessible.
**Discoverability.** After this change, `exists(options, ...)` no longer tells you from syntax alone whether `options` is a relationship or an embedded array. This is the intended outcome (unification is the point), but a small introspection helper such as `Ash.Resource.Info.collection_kinds/1` could soften the debugging experience. Mentioned only as a follow-up; not part of this proposal.
**Origin.** This came up while implementing a "search across snapshot and line items" feature on a project where the same logical concept (a list of items belonging to a parent) is modeled as `has_many` in some places and as an embedded array in others depending on whether snapshotting semantics are needed. Today the two cases require fundamentally different expression code; this proposal makes them uniform.
**Implementation.** Happy to draft the PR (Ash core → `Filter.Runtime` → AshPostgres) once the seven design decisions above are accepted or revised.
Contributor guide
Research direction
Start with expand_through_path/3 in lib/ash/filter/filter.ex and Ash.Query.Exists, then trace the Exists branch in do_hydrate_refs and the matching paths in Ash.Filter.Runtime and Ash.Filter.match?/2. Confirm the segment representation and capability behavior with maintainers before implementing. Done means relationship, embedded-array, nested, mixed-path, and in-memory filtering cases compose through exists/2 without changing existing relationship behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100