GenericMappingTools / GenericMappingTools/gmt
Slow processing of large, compressed netCDF files tied to a small chunk size
- Dominant language
- C
- Stars
- 979
- Forks
- 414
- Avg merge
- 17h 26m
- Merged PRs (30d)
- 54
Description
I created a large global 1mx1m compressed (`--IO_NC4_DEFLATION_LEVEL=9`) netCDF file from one of the Sandwell/Smith IMG files; dimensions are 21600x17280. We noticed it took "forever" to get gmt grdinfo **-M** to report the min/max values (over 1 minute on my iMac) and gdalinfo took many minutes; yet this varied widely among Windows and other installations. [Initial investigations](https://lists.osgeo.org/pipermail/gdal-dev/2020-January/051389.html) determined the culprit to be a small default cache size setting when netCDF library is built [those who built their own netCDF library with a larger cache got much better response], but today I noticed that if I recreate the grid using `--IO_NC4_CHUNK_SIZE=4096` the speed is back to normal. So, I am wondering if our default selection of chunk sizes (which seems to be in the 128-256 range) is just way too small for at least large grids like this one. Perhaps the _gmtnc_set_optimal_chunksize_ function needs some more scrutiny. Not sure if @ldldr is listening but would like his opinion.
Contributor guide
Assessment
This issue has not been assessed yet.