(very complicated) ROMS_SWAN_nesting blow up issue

Nobody has claimed this yet.

I am new to COAWST. I am now working on a project: ROMS_SWAN coupled with nesting (4 nested layers in total).

ROMS and SWAN are sharing the same grid and there are 4 nested layers in total. The largest domain covers -5 to 50 degree in latitude and 100 to 150 degree in longitude.

The complicated point is that when i run ROMS and SWAN separately, they run smoothly (144 forecast hours in total) without any blowup. I have also took a look at the output files from these two models very carefully (like the qck file for ROMS and swan netCDF files converted from the mat files) and everything is quite normal for ALL the 4 nested layers. (there isn't any distinct points with extreme values).

However, when i run the coupled one: ROMS_SWAN, it blew up after 1 forecast hour. When i take a look at the output QCK file for ROMS, i can notice that in the bottom left corner of the largest domain, the zeta values are horribly large, reaching almost 100 metres after 1 forecast hour, and the u & v are also large (increasing rapidly to 20 metres per second before blowing up. Another thing that i noticed is that, beside the horrible point above, there is also a small region near my SECOND largest domain's boundary (boundary again). When i take a look at the largest domain's qck file, actually in the corresponding region, the values are
quite large already compared with other points (like the region points are having 2-3 metre per second while other "normal" points are having -1 to 1 in general). As this region is located near the boundary of the second largest domain, so i think that's the reason why they are somehow exaggerated 3-4 times, having up to 8-10 metre per second in the second largest domain.

I have done quite a lot searching regarding this matter via the internet, and i tried using the following methods:

1. masking (masking those grid points with large zeta, u and v)
--> Result: when i mask the "blew up point", yes it can run longer for few minutes, then this time this is other point blew up (i think actually those points' u and v speed values are increasing continuously from the beginning, it does not blow up in the previous run because other points blew up BEFORE them). But one finding is that those points are mainly located near the boundary of the nested layer.

2. adding sponge layer near the boundary: increasing the viscosity and diffusity near the boundaries
--> Result: No big difference at all, still blew up

3. Smoothing the bathymetry
I did this all because i noticed that in those blew up points, all they have in common apart from near the boundary is that they are having very steep bathymetry, like near the coast of the Philippines and Ryukyu islands. I tried to smoothen the bathymetry using matlab scripts.

--> Result: Yes it can prevent some points from blowing up, but still, there are other points blowing up (especially near the Ryukyu islands, as i think they are still having quite steep bathymetry). But the point is that i already smoothened them to a large extent (using rx0 = 0.04).

What i dont understand is that:

A. May I know why when i separately run them, they are all ok without any abnormal values, but when i coupled them, it suddenly blew up?

B. As i suspect it might be related to the coupling, so i tried to increase the coupling interval between the two models by changing the coupling*.in (from 1800(s) to 18000(s) in both OCE2WAV AND WAV2OCE). I thought it could reduce the exchange of info between the two models (please correct me if i am wrong, as i am new to COAWST). However, still, it runs till the 1st forecast hour and then blew up. I dont understand, suppose if the coupling interval is 18000 seconds (they cannot exchange stuffs at all, but still the same result)

I will be very grateful if anyone could share their experiences if they encountered similar issues or give me some hints. Thank you very much

Start with the coupled ROMS_SWAN configuration and the coupling*.in files, especially the OCE2WAV and WAV2OCE intervals, then compare the standalone and coupled qck and SWAN NetCDF outputs around the nested-layer boundaries. Review the MATLAB bathymetry-smoothing setup and reproduce the one-hour blowup; done means identifying a reproducible coupling or boundary cause and documenting a stable configuration.

Written by the indexing model from the issue text.