spcl / spcl/dace

Reproducer For Nondeterministic Transformation Order

Open
#2,181 0 comments 0 reactions 1 assignee Claimed by @acalotoiu View on GitHub
Dominant language
Python
Stars
593
Forks
163
Avg merge
2d 23h
Merged PRs (30d)
60

Description

In GT4Py we found a case were the order of transformations, or in which order the patterns are fed to the transformation are non deterministic.
In our case this leads to an SDFG with more memory consumption and larger runtime.

Image

Here you see an image of the case where the SDFG has taken the wrong turn.
As you can see there is this `lambda_2___tmp0_0_split_0` AccessNode that connects two Maps of equal range.
In the "good/desirable" version, these two Maps would be partially fused.
This did not happen because for some reason something was merged into the top Map that prevents the fusing to start.

## Steps To Reproduce

### Installing
Firs of all you have to install ICON4Py.
I have provided a version that you can use, please clone from `https://github.com/philip-paul-mueller/icon4py` and check out the branch `__dace_issue_2181__reproducer`.
Then follow the instruction and install ICON4Py, which essentially boils down to run `uv sync --extra all`.

Now you have to install GT4Py.
For this you first have to uninstall the version that was installed by ICON4Py.
After that you should clone GT4Py from `https://github.com/philip-paul-mueller/gt4py/` and checkout `__dace_issue_2181__reproducer`.
Now install it in editable mode, for this run `uv pip install -e . ` in the cloned repo.

Now you have to install the right version of DaCe, since current GT4Py does not run with vanilla DaCe (maybe in the meantime it would work, I have to check it).
First check if DaCe was installed by either ICON4Py or GT4Py and uninstall it if it is present.
Then do the same as for the two other things, clone from `https://github.com/GridTools/dace` and check out `__gt4py-next-integration_2025_09_25` (it is a tag).
After that install it again in editable mode by running `uv pip install -e . `.

### Running
Go into the ICON4Py folder, to run the stencil that causes problems use the following command:
```
GT4PY_COLLECT_METRICS_LEVEL=10 GT4PY_BUILD_CACHE_LIFETIME=PERSISTENT pytest -m continuous_benchmarking --benchmark-only -sv --backend=dace_cpu --grid=icon_benchmark -k 'test_TestFusedVelocityAdvectionStencilVMomentumContinuousBenchmarking[compile_time_domain]' model/atmosphere/dycore/tests/dycore/stencil_tests/test_compute_advection_in_vertical_momentum_equation.py
```

This command will run the pipeline and benchmark it.
You can find the SDFG inside GT4Py's cache folder that is located in `${PWD}/.gt4py_cache`, look inside and you should find the folder created by DaCe.
The two SDFG manifests almost 50:50 so you might have to run the command above some times to get the good and the bad one (see the notes before doing it).

Note the following:
- This will run on CPU, if you want to run on GPU replace `dace_cpu` with `dace_gpu`.
- The first time, it will download the grids and test data it needs, this means it will take some time.
- If you run the command again it will not be lowered and processed, instead the cached SDFG is used.
To remove the cache use `rm -rf .gt4py_cache`.
- The interesting part of the pipeline for you starts [here](https://github.com/philip-paul-mueller/gt4py/blob/76a5484c844a94b77d2988e7c845bc441e6f3e75/src/gt4py/next/program_processors/runners/dace/workflow/translation.py#L365).
- GT4Py's auto optimizer is located [here](https://github.com/philip-paul-mueller/gt4py/blob/76a5484c844a94b77d2988e7c845bc441e6f3e75/src/gt4py/next/program_processors/runners/dace/transformations/auto_optimize.py#L48).

# Attachments
- [FAST_program.sdfg.gz](https://github.com/user-attachments/files/22964312/FAST_program.sdfg.gz)
- [SLOW_programm.sdfg.gz](https://github.com/user-attachments/files/22964324/SLOW_programm.sdfg.gz)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.