AMReX-Astro / AMReX-Astro/Castro
Can we write the advance algorithm the way non-subcycling AMR algorithms look?
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 340
- Forks
- 105
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 8
Description
A non-subcycling AMR version of our code (in the Strang build, not the simplified SDC build) would do the following (assumes two levels for simplicity, and ignores burning for now):
(1) Compute old-time coarse-level source predictor (first-order accurate) and apply to state
(2) Compute second-order accurate hydro source on coarse level
(3) Do old-time Strang burn on fine-level (directly updates the state)
(4) Compute old-time fine-level source predictor and apply to state
(5) Compute hydro source on fine level and apply to state
(6) Correct coarse level hydro source with fine level hydro source
(7) Compute new-time fine-level source corrector and apply to state (makes the advance fully second-order accurate)
(8) Apply hydro source on coarse level
(9) Compute new-time coarse-level source corrector and apply to state
For example, this is what the FLASH paper suggests. But Castro does (8) and (9) before the fine-level, and then correct the hydro and non-hydro source terms on the coarse level during the reflux operation. The primary downside of our approach is that it involves a re-calculation of the source terms on the coarse level. If the source terms are simple, this is cheap, but when we are using gravity, the cost of the correction solve on gravity can be expensive. Section 6.2.3 of the Castro paper comments that this is not so bad because the correction solve should be easier than the initial level solve. However, the gravity solve is so expensive on GPUs relative to everything but burning that having three solves per level rather than two really does pose a serious cost.
Berger and Colella (1989) use subcycling but agree with this modification, emphasizing that the flux correction should be done as a separate step after both the fine grids and coarse grids are done with the main advance. But they were describing a pure hydrodynamics algorithm without sources, so from a cost perspective it was probably fairly neutral (the flux correction operation for hydro alone is cheap). The question is: can we have steps 3-7 be done for multiple subcycles on the fine level before steps 8-9? After all, Berger and Colella's argument is only that doing it in this modified order makes the code simpler, which is an aesthetic consideration, not a physics/math consideration.
The argument for why we need to do it this way when subcycling is that we may need to collect information from the coarse grid during the fine grid advance. For example, a FillPatch operation on the fine grid will borrow data interpolated in time from the coarse grid on the boundaries of the fine grid, or a regrid during the fine timesteps may interpolate from the coarse grid to fill the data. So we need to have a valid estimation of U^{n+1} on the coarse grid while doing the fine steps.
Given that, is there anything we can do to avoid calculating the source terms three times per step? Idea/proposal to follow in a subsequent comment.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the Strang-build advance path and the ordering of fine-level subcycles, coarse-level updates, FillPatch, regridding, refluxing, and source solves. Determine whether steps 3–7 can precede steps 8–9 while keeping coarse boundary data valid, and whether this avoids the extra gravity/source calculation without changing the algorithm's correctness.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100