JuliaDataCubes / JuliaDataCubes/EarthDataLab.jl
saving a GDALCube from multiple vrts fails with distributed computing
Open
@felixcremer is already working on this.
Since Feb 7, 2022.
- Dominant language
- Julia
- Stars
- 34
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
When I try to use savecube for a cube generated from nested VRTs with the gdalcubes function from ESDLHelpers this works for multi threaded code, but it fails, when I am using Distributed and SlurmClusterManager with the following
Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n") [1]
runLoopError during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")(
::Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")ESDL.DAT
.Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")DATConfig
{Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")1
,Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n")1
}Error during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n"),
::BoolError during running loop: GDAL.GDALError(GDAL.CE_Failure, 10, "Pointer 'hDS' is NULL in 'GDALDatasetRasterIOEx'.\n"))
at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:408
[2] mapCube(::ESDL.Proc.var"#cop#30", ::Tuple{ESDLArray{Float32,3,ArchGDAL.RasterDataset{Float32,ArchGDAL.Dataset},Array{CubeAxis,1}}}; max_cache::Float64, indims::InDims, outdims::OutDims, inplace::Bool, ispar::Bool, debug::Bool, include_loopvars::Bool, showprog::Bool, nthreads::Dict{Int64,Int64}, loopchunksize::Dict{Any,Any}, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:278
[3] #mapCube#34 at /home/crem_fe/.julia/dev/ESDL/src/DAT/DAT.jl:177 [inlined]
[4] savecube(::ESDLArray{Float32,3,ArchGDAL.RasterDataset{Float32,ArchGDAL.Dataset},Array{CubeAxis,1}}, ::String; chunksize::Dict{Any,Any}, max_cache::Float64, backend::Symbol, backendargs::Base.Iterators.Pairs{Symbol,Integer,Tuple{Symbol,Symbol},NamedTuple{(:fill_value, :overwrite),Tuple{Int64,Bool}}}) at /home/crem_fe/.julia/dev/ESDL/src/Proc/CubeIO.jl:38
[5] top-level scope at REPL[12]:1
I'd have to further check, whether this also fails on normally distributed usage.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.