AMReX-Astro / AMReX-Astro/Castro

Can we write the advance algorithm the way non-subcycling AMR algorithms look?

Open
#2,183 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

driver study
Dominant language
C++
Stars
340
Forks
105
Avg merge
3d 8h
Merged PRs (30d)
8

Description

A non-subcycling AMR version of our code (in the Strang build, not the simplified SDC build) would do the following (assumes two levels for simplicity, and ignores burning for now):

(1) Compute old-time coarse-level source predictor (first-order accurate) and apply to state
(2) Compute second-order accurate hydro source on coarse level

(3) Do old-time Strang burn on fine-level (directly updates the state)
(4) Compute old-time fine-level source predictor and apply to state
(5) Compute hydro source on fine level and apply to state
(6) Correct coarse level hydro source with fine level hydro source
(7) Compute new-time fine-level source corrector and apply to state (makes the advance fully second-order accurate)

(8) Apply hydro source on coarse level
(9) Compute new-time coarse-level source corrector and apply to state

For example, this is what the FLASH paper suggests. But Castro does (8) and (9) before the fine-level, and then correct the hydro and non-hydro source terms on the coarse level during the reflux operation. The primary downside of our approach is that it involves a re-calculation of the source terms on the coarse level. If the source terms are simple, this is cheap, but when we are using gravity, the cost of the correction solve on gravity can be expensive. Section 6.2.3 of the Castro paper comments that this is not so bad because the correction solve should be easier than the initial level solve. However, the gravity solve is so expensive on GPUs relative to everything but burning that having three solves per level rather than two really does pose a serious cost.

Berger and Colella (1989) use subcycling but agree with this modification, emphasizing that the flux correction should be done as a separate step after both the fine grids and coarse grids are done with the main advance. But they were describing a pure hydrodynamics algorithm without sources, so from a cost perspective it was probably fairly neutral (the flux correction operation for hydro alone is cheap). The question is: can we have steps 3-7 be done for multiple subcycles on the fine level before steps 8-9? After all, Berger and Colella's argument is only that doing it in this modified order makes the code simpler, which is an aesthetic consideration, not a physics/math consideration.

The argument for why we need to do it this way when subcycling is that we may need to collect information from the coarse grid during the fine grid advance. For example, a FillPatch operation on the fine grid will borrow data interpolated in time from the coarse grid on the boundaries of the fine grid, or a regrid during the fine timesteps may interpolate from the coarse grid to fill the data. So we need to have a valid estimation of U^{n+1} on the coarse grid while doing the fine steps.

Given that, is there anything we can do to avoid calculating the source terms three times per step? Idea/proposal to follow in a subsequent comment.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the Strang-build advance path and the ordering of fine-level subcycles, coarse-level updates, FillPatch, regridding, refluxing, and source solves. Determine whether steps 3–7 can precede steps 8–9 while keeping coarse boundary data valid, and whether this avoids the extra gravity/source calculation without changing the algorithm's correctness.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.