Parameter Space error
- Dominant language
- Julia
- Stars
- 66
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
Our fluxes computation requires a lot of parameter space and fails on older GPUs when using a tripolar grid.
MWE: using both `Oceananigans#main` and `OrthogonalSphericalShellGrids#main`
```julia
using Oceananigans
using ClimaOcean
using OrthogonalSphericalShellGrids
arch = GPU()
grid = TripolarGrid(arch;
size = (50, 50, 10),
halo = (7, 7, 7),
z = collect(-6000:600:0),
first_pole_longitude = 75,
north_poles_latitude = 55)
bottom_height = retrieve_bathymetry(grid;
minimum_depth = 10,
dir = "./",
interpolation_passes = 20,
connected_regions_allowed = 0)
grid = ImmersedBoundaryGrid(grid, GridFittedBottom(bottom_height); active_cells_map = true)
free_surface = SplitExplicitFreeSurface(grid; substeps = 20)
ocean = ocean_simulation(grid; free_surface)
model = ocean.model
backend = JRA55NetCDFBackend(4)
atmosphere = JRA55_prescribed_atmosphere(arch; backend)
radiation = Radiation(arch)
sea_ice = ClimaOcean.OceanSeaIceModels.MinimumTemperatureSeaIce()
coupled_model = OceanSeaIceModel(ocean, sea_ice; atmosphere, radiation)
```
On a Titan V it leads to the code failing with this error
```julia
**julia> coupled_model = OceanSeaIceModel(ocean; atmosphere, radiation)
ERROR: Kernel invocation uses too much parameter memory.
4.914 KiB exceeds the 4.000 KiB limit imposed by sm_70 / PTX v7.8.
Relevant parameters:
[1] __ctx__::KernelAbstractions.CompilerMetadata{Oceananigans.Utils.OffsetStaticSize{(0:51, 0:51)}, KernelAbstractions.NDIteration.DynamicCheck, Nothing, Nothing, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.StaticSize{(4, 4)}, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, Tuple{Int64, Int64}, Oceananigans.Utils.KernelOffsets{Tuple{Int64, Int64}}}} uses 32 bytes
[2] similarity_theory::SimilarityTheoryTurbulentFluxes{Float64, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.SimilarityScales{ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.MomentumStabilityFunction{Float64}, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.ScalarStabilityFunction{Float64}, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.ScalarStabilityFunction{Float64}}, ClimaOcean.OceanSeaIceModels.PrescribedAtmospheres.PrescribedAtmosphereThermodynamicsParameters{Float64}, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.ClasiusClapyeronSaturation, Float64, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.SimilarityScales{ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.MomentumRoughnessLength{Float64, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.TemperatureDependentAirViscosity{Float64}}, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.ScalarRoughnessLength{Float64, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.TemperatureDependentAirViscosity{Float64}, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.ReynoldsScalingFunction{Float64}}, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.ScalarRoughnessLength{Float64, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.TemperatureDependentAirViscosity{Float64}, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.ReynoldsScalingFunction{Float64}}}, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.LogarithmicSimilarityProfile, ClimaOcean.OceanSeaIceModels.CrossRealmFluxes.RelativeVelocity, @NamedTuple{latent_heat::Field{Center, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, sensible_heat::Field{Center, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, water_vapor::Field{Center, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, x_momentum::Field{Center, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, y_momentum::Field{Center, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}}} uses 984 bytes
[3] grid::ImmersedBoundaryGrid{Float64, Periodic, RightConnected, Bounded, OrthogonalSphericalShellGrid{Float64, Periodic, RightConnected, Bounded, OffsetArrays.OffsetMatrix{Float64, CUDA.CuDeviceMatrix{Float64, 1}}, OffsetArrays.OffsetVector{Float64, CUDA.CuDeviceVector{Float64, 1}}, OffsetArrays.OffsetVector{Float64, CUDA.CuDeviceVector{Float64, 1}}, OrthogonalSphericalShellGrids.Tripolar{Int64, Int64, Int64}, Nothing}, GridFittedBottom{Field{Center, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, Oceananigans.ImmersedBoundaries.CenterImmersedCondition}, CUDA.CuDeviceVector{Tuple{UInt8, UInt8, UInt8}, 1}, CUDA.CuDeviceVector{Tuple{UInt8, UInt8}, 1}, Nothing} uses 1.469 KiB
[4] clock::@NamedTuple{time::Float64, last_Δt::Float64, last_stage_Δt::Float64, iteration::Int64, stage::Int64} uses 40 bytes
[5] ocean_state::@NamedTuple{u::SubArray{Float64, 2, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Tuple{Base.Slice{OffsetArrays.IdOffsetRange{Int64, Base.OneTo{Int64}}}, Base.Slice{OffsetArrays.IdOffsetRange{Int64, Base.OneTo{Int64}}}, Int64}, true}, v::SubArray{Float64, 2, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Tuple{Base.Slice{OffsetArrays.IdOffsetRange{Int64, Base.OneTo{Int64}}}, Base.Slice{OffsetArrays.IdOffsetRange{Int64, Base.OneTo{Int64}}}, Int64}, true}, w::SubArray{Float64, 2, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Tuple{Base.Slice{OffsetArrays.IdOffsetRange{Int64, Base.OneTo{Int64}}}, Base.Slice{OffsetArrays.IdOffsetRange{Int64, Base.OneTo{Int64}}}, Int64}, true}, T::SubArray{Float64, 2, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Tuple{Base.Slice{OffsetArrays.IdOffsetRange{Int64, Base.OneTo{Int64}}}, Base.Slice{OffsetArrays.IdOffsetRange{Int64, Base.OneTo{Int64}}}, Int64}, true}, S::SubArray{Float64, 2, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Tuple{Base.Slice{OffsetArrays.IdOffsetRange{Int64, Base.OneTo{Int64}}}, Base.Slice{OffsetArrays.IdOffsetRange{Int64, Base.OneTo{Int64}}}, Int64}, true}, e::SubArray{Float64, 2, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Tuple{Base.Slice{OffsetArrays.IdOffsetRange{Int64, Base.OneTo{Int64}}}, Base.Slice{OffsetArrays.IdOffsetRange{Int64, Base.OneTo{Int64}}}, Int64}, true}} uses 768 bytes
[7] atmos_state::@NamedTuple{u::OffsetArrays.OffsetArray{Float32, 4, CUDA.CuDeviceArray{Float32, 4, 1}}, v::OffsetArrays.OffsetArray{Float32, 4, CUDA.CuDeviceArray{Float32, 4, 1}}, T::OffsetArrays.OffsetArray{Float32, 4, CUDA.CuDeviceArray{Float32, 4, 1}}, q::OffsetArrays.OffsetArray{Float32, 4, CUDA.CuDeviceArray{Float32, 4, 1}}, r::OffsetArrays.OffsetArray{Float32, 4, CUDA.CuDeviceArray{Float32, 4, 1}}, p::OffsetArrays.OffsetArray{Float32, 4, CUDA.CuDeviceArray{Float32, 4, 1}}} uses 528 bytes
[8] atmos_grid::Oceananigans.Grids.ZRegularLLG{Float32, Periodic, Bounded, Flat, OffsetArrays.OffsetMatrix{Float32, CUDA.CuDeviceMatrix{Float32, 1}}, OffsetArrays.OffsetVector{Float32, CUDA.CuDeviceVector{Float32, 1}}, OffsetArrays.OffsetVector{Float32, CUDA.CuDeviceVector{Float32, 1}}, OffsetArrays.OffsetVector{Float32, CUDA.CuDeviceVector{Float32, 1}}, Float32, OffsetArrays.OffsetVector{Float32, CUDA.CuDeviceVector{Float32, 1}}, OffsetArrays.OffsetVector{Float32, CUDA.CuDeviceVector{Float32, 1}}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, Nothing} uses 1024 bytes
[9] atmos_times::StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64} uses 48 bytes
[10] atmos_backend::JRA55NetCDFBackend uses 16 bytes
[11] atmos_time_indexing::Oceananigans.OutputReaders.Cyclical{Float64} uses 8 bytes
[12] atmosphere_reference_height::Float32 uses 4 bytes
[13] atmosphere_boundary_layer_height::Float32 uses 4 bytes
[14] atmos_thermodynamics_parameters::ClimaOcean.OceanSeaIceModels.PrescribedAtmospheres.PrescribedAtmosphereThermodynamicsParameters{Float32} uses 56 bytes
Note: use a newer CUDA to support more parameters on your device.**
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the provided MWE and the OceanSeaIceModel(ocean; atmosphere, radiation) construction on the Titan V. Inspect the reported CUDA kernel parameter list, especially similarity_theory and grid, and trace the fluxes computation that triggers the error. Done means the same construction no longer exceeds the device's parameter-memory limit on the affected GPU.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100