ClimaCore remaining refactor (including v1)

# ClimaCore v1.0 Roadmap: Path to Release with CG and DG

## Context & Purpose

Succeeds #2554 and #2468. Tracks the remaining work to release **ClimaCore v1.0**:
a stable, unified, performant dynamical-core infrastructure supporting both
**Continuous Galerkin (CG)** and **Discontinuous Galerkin (DG)** horizontal
discretizations, with vertical finite-difference infrastructure.

## Definition of "v1.0"

Carried forward from #2554, these are the release *gates*:

1. **API frozen.** Every exported symbol is one we are prepared to support
through v1.x; deprecation shims are removed or explicitly documented.
2. **Correctness.** CG and DG pass conservation and CPU/GPU-equivalence tests on
cubed-sphere and extruded spaces, single- and multi-rank, under
`--check-bounds=yes`.
3. **Performance.** No allocation or significant launch-latency regressions
against the current benchmarks; masked and fused paths covered.
4. **Docs.** Architecture docs following Diataxis principles.
5. **Downstream green.** ClimaAtmos, ClimaLand, ClimaCoupler, ClimaDiagnostics,
ClimaUtilities pass against the release.

---

## Merged Since #2554

- [x] **Phase 1 — DataLayouts unification.** 4 core layout types, universal
`(v, i, j, h)` indexing, `DataScope`/`foreach_slice` loop abstractions (#2522).
- [x] **Field views in `foreach_slice`**, letting Field-level broadcasts fuse
through the DataLayout loop primitives (#2571). Related GPU hardening: the
inference-widening fix for broadcast-argument traversal (#2575) and linear
striding / singleton broadcasting in GPU indexing (#2568).
- [x] **Phase 2 — FD operator stencil abstraction.** Operator deduplication,
interior/boundary stencil consolidation, linear operators as matrix
multiplies (#2544, #2526, #2548).
- [x] **Phase 3.2 — Strong/weak spectral form unification** via type-parameter
dispatch (#2556).
- [x] **Test & example consolidation** (#2570, #2576).
- [x] **DG infrastructure (initial).**
- Spaces can be marked discontinuous (`discontinuous = true` at grid
construction); `weighted_dss!` becomes a no-op on them, and the flag
round-trips through HDF5 serialization (#2599).
- Horizontal DG face operators with CUDA kernels: `CentralNumericalFlux` and
`RusanovNumericalFlux`, interior and boundary evaluators, lifting functions
for non-conservative terms, and an LDG/interior-penalty scalar Laplacian
(`LDGLaplacianFlux`), with conservation and stability unit tests on plane,
sphere, and extruded spaces (#2602).
- Distributed MPI halo exchange restricted to element faces
(`start_dg_ghost_exchange` with a shared `DGGhostExchange` handle), plus
distributed DG tests at 2 and 3 ranks that verify cross-rank conservation
to roundoff (#2603).

---

## Merged on 8/31/26:

### Phase 3.0/3.1 — Spectral element operator fusion — #2598

**This is the current bottleneck for subsequent work.** It rewrites `SpectralBroadcasted` as
a `Broadcasted{SpectralStyle}`, so spectral operators compose with
pointwise broadcasts and run through the same `foreach_slice` slab-loop
machinery on CPU and GPU, making it possible to delete the hand-written CUDA kernels
(`operators_spectral_element.jl`, `operators_sem_shmem.jl`).

Everything downstream of Phase 3 in the sequencing below assumes #2598 has
landed.

### GPU performance optimizations — #2606 (`ts/perf`, stacks on #2598)

+28.7% speedup on the A100 baroclinic wave: FieldVector broadcasts flatten
to GPU linear indexing without aliasing copies, launch/occupancy tuning,
coalesced DG face kernels (1.72× faster face fluxes, 30% overall speedup), and
the hyperdiffusion Laplacian atoms `Operators.scalar_laplacian` / `vector_laplacian`
make the hybrid hyperdiffusion example discretization-agnostic.

### Compile time (TTFX) — #2607 (`ts/ttfx`, stacks on #2606)

A dependency-free precompilation workload
(`CLIMA_SKIP_PRECOMPILE_WORKLOAD=true` opts out) plus a modularized
`_SpectralElementGrid2D`: representative first-use latency drops
16.9 s → 4.5 s. See §8 for what remains.

---

## Remaining Work

Rough sizes: **S** = one focused PR, **M** = a few PRs, **L** = multi-PR effort.

### 1. Boundary-condition and DG naming — [size: M]

*Harmonize naming across FD, CG, DG before the API freezes.

- [x] FD advection outflow [@tapios, done in #2608]: keep `Extrapolate{N}` as the numerical primitive
(the name says what it does). If a physically named entry point is wanted, add a
documented convenience constructor `Outflow(; order = 0)` that
returns `Extrapolate{order}()`, and let DG boundary outflow share it. Keep
FD-only momentum concepts (free-slip, no-slip) separate from DG
numerical-flux naming.
- [x] DG assembly API [@tapios, done in #2608]: rename `internal` → `interior` to match standard DG
terminology and pair with `boundary`:
`add_numerical_flux_internal!` → `add_numerical_flux_interior!` and
`add_lifting_flux_internal!` → `add_lifting_flux_interior!`.
- [x] Downstream PRs in ClimaAtmos and ClimaLand migrating to new CC: E.g., replace
`FirstOrderOneSided()` → `Extrapolate()` and `ThirdOrderOneSided()` → `Extrapolate(1)`:
ClimaAtmos (2 sites), ClimaLand (2 sites). More complete draft list in issue #2619.
Downstream needs to re-baseline regression tests [@imreddyTeja].

### 2. UnrolledUtilities integration — [size: S] [@dennisYatunin]

- [x] Resolve index-typing discrepancies (`Val{i}`, `Int`, `StaticOneTo`,
`CartesianIndex`) and unrolled-slice bounds/Cartesian conversion in
UnrolledUtilities.jl [@dennisYatunin, done in CliMA/UnrolledUtilities.jl#36]
- [x] Move ClimaCore's local shims upstream (or remove them):
`unrolled_setindex`, `unrolled_insert`, `unrolled_map_with_inbounds`
(see the `module Unrolled # TODO` in `src/Utilities/Utilities.jl`)
[@dennisYatunin, done in CliMA/UnrolledUtilities.jl#36]
- [ ] Release UU 0.1.11; bump the `UnrolledUtilities` pin in `Project.toml` from
`0.1.9` to `0.1.11; delete the local shims. Drop the source-text guardrail in
`test/aqua.jl`, replacing it with GPU IR validation tests in UnrolledUtilities.
- [ ] Extend GPU tests from UnrolledUtilities to ClimaCore, adding tests asserting
that component names, basis projections, and unrolled index selection fold into
type parameters without throwing IR errors during compilation.

### 3. Column loop primitives — [size: M] [@dennisYatunin]

- [ ] **Serial-per-column launches must be sized from the column count.**
The originally described waste (a column loop launching `Nv × Ncols` threads
with `Nv − 1` of every `Nv` idle) no longer exists: #2598's `ThisSubBlock`
launch path now sizes column/slab/level loops from slice counts, and the
remaining point-count path in `foreach_slice(::ThisHost, ...)` only
handles one-point (`view`) slices, where point count equals slice count.
What survives is a requirement on the primitives below: `foreach_column` /
`column_accumulate!` need one thread traversing a whole column serially — a
mode that does not exist yet. That launch must be sized from the number
of columns. Optionally add a `gpu_cuda_threadblocks.jl`-style assertion
pinning slice-count-derived launch configurations.
- [x] **Define `foreach_column` (and `foreach_slab`, `foreach_level`).** The
docstrings in `src/DataLayouts/scopes.jl` already cross-reference these as
`@ref`s, but only `foreach_slice(column, f, args...)` exists — add the
wrappers or fix the dangling references.
- [ ] Implement `column_accumulate!` (prefix sums / vertical integrals) beside
the existing `column_reduce!`; support masking and `flip = Val(true)` reverse
traversal. The same TODO in `src/DataLayouts/loops.jl` also names
`column_stencil!` and `slab_convolve!` (related: #2530, the convolution
primitive).
- [ ] Document the hardware limitations: `foreach_column` suits memory-bound,
thread-independent column work; tall serial vertical stencils (Thomas solves)
belong in cooperative block-level PCR in shared memory, not here.

> **Dropped: PCR `ldiv!` fusion** (#2554 Phase 6). It is worth trying again with
> column fusion but unlikely to yield substantial benefit.

### 4. DG/CG parity and integration — [size: L] [@akshaysridhar]

*The largest remaining bucket. Builds on the merged DG work (#2599/#2602/#2603)
and on Phase 3 fusion.*

- [x] DG conservation CI on GPU and at scale. Single-rank conservation tests and
2-/3-rank CPU conservation-to-roundoff tests are merged; still missing are
distributed-GPU runs (the current distributed test emulates the GPU ghost
algorithm on host and skips real GPUs) and CPU-vs-GPU equivalence for every
face kernel.
- [x] **Model-level CG↔DG switching.** [done in #2609] The space-level switch exists
(`discontinuous = true` at grid construction; `Spaces.is_continuous` makes
DSS a no-op), and #2606's
Laplacian atoms demonstrate the operator-level pattern (the hybrid hyperdiffusion
example now runs one prep → `weighted_dss!` → apply sequence on both
discretizations). What remains is switching a *model* by configuration: tendency
assembly must swap `weighted_dss!` for the numerical-flux calls without
restructuring state `FieldVector`s or tendencies. Note `Operators.vector_laplacian`
still errors on DG spaces (needs grad-div/curl-curl face lifting, below).
- [x] Viscous/diffusive terms beyond the merged LDG/interior-penalty scalar
Laplacian: tensor viscous stresses, and BR1/BR2 alternatives if the
interior-penalty form proves insufficient.
- [x] Optimize interior/boundary face kernels (`ext/cuda/operators_dg.jl`,
assembly in `src/Operators/numericalflux.jl`) for coalesced gather/scatter
across element faces [@tapios, done in #2606: level-first staging and
surface-geometry layouts, `(v, q, f)` thread order, 1.72× faster face
fluxes].

### 5. Type aliases and API freeze — [size: M] [@imreddyTeja]

- [ ] Fold `Device*` grid variants (`DeviceFiniteDifferenceGrid`,
`DeviceExtrudedFiniteDifferenceGrid`, `DeviceSpectralElementGrid1D/2D`,
`DeviceIntervalTopology`) into `Adapt.adapt_structure` on the main types,
eliminating the parallel hierarchy that can silently go stale. #2631
- [ ] Remove legacy `Field` aliases. `ColumnField` exists solely because
ClimaAtmos uses it in type annotations (see the comment in
`src/Fields/Fields.jl`) — removal requires a coordinated ClimaAtmos PR. See #2619
- [ ] Simplify space constructors (carried over from #2554 Phase 4.4): remove
deprecated constructors, one keyword entry point per space type, grid
construction delegated to `Grids` exclusively. See #2619
- [ ] Freeze and document the exported surface (API gate).

### 6. Documentation — [size: M] [@tapios, done in #2622]

- [x] Architecture docs: CG spectral elements, DG face formulation, staggered FD
vertical stencils; the `DataLayouts` memory model, `DataScope`, loop
primitives. Fix docstring cross-references to not-yet-defined loop wrappers
(or define them, §3).
- [x] Canonical verified examples: 1D column (advection–diffusion), 2D plane
(waves, CG & DG), 3D sphere (baroclinic wave, CG & DG (**DG TBD once we have tensor divergence**).

### 7. Release engineering — [size: S] [@imreddyTeja]

- [ ] Downstream compat gates: ClimaAtmos, ClimaLand, ClimaCoupler,
ClimaDiagnostics, ClimaUtilities.
- [ ] Tag v1.0, once all five release gates hold.

---

### 8. Compile time (TTFX) — [size: M]

*Follow-ups to #2607. Independent of everything
else on this roadmap.*

- [ ] Apply the `_SpectralElementGrid2D` modularization treatment to the
`ExtrudedFiniteDifferenceGrid` constructor [@imreddyTeja].
- [ ] Audit downstream invalidations (one invalidated common method silently
undoes the workload's savings). [@imreddyTeja]

## Sequencing

Phase 3 fusion (#2598) gates the DG/CG parity work and the API freeze; #2606
stacks on #2598 and `ts/ttfx` stacks on #2606. The §2 UnrolledUtilities work
and the §8 compile-time work are independent of each other and of #2598.

```mermaid
flowchart TD
P3["#2598 Spectral fusion (fix GPU + 1.11 failures, merge)"]
UU["UnrolledUtilities 0.2 + shim upstreaming (§2)"]
COL["Column loop primitives (§3)"]
NAME["BC / DG naming (§1)"]
DG["DG/CG parity + conservation CI (§4)"]
API["Type-alias → trait cleanup + API freeze (§5)"]
DOCS["Docs + migration guide (§6)"]
REL["Downstream validation + release (§7)"]

P3 --> DG
P3 --> API
NAME --> DG
DG --> API
API --> DOCS
UU --> API
COL --> DG
DOCS --> REL
DG --> REL
REL --> V1["ClimaCore v1.0"]
```