GenericMappingTools / GenericMappingTools/gmt

Slow processing of large, compressed netCDF files tied to a small chunk size

Open
#2,402 12 comments 0 reactions 0 assignees View on GitHub
longterm
Dominant language
C
Stars
979
Forks
414
Avg merge
17h 26m
Merged PRs (30d)
54

Description

I created a large global 1mx1m compressed (`--IO_NC4_DEFLATION_LEVEL=9`) netCDF file from one of the Sandwell/Smith IMG files; dimensions are 21600x17280. We noticed it took "forever" to get gmt grdinfo **-M** to report the min/max values (over 1 minute on my iMac) and gdalinfo took many minutes; yet this varied widely among Windows and other installations. [Initial investigations](https://lists.osgeo.org/pipermail/gdal-dev/2020-January/051389.html) determined the culprit to be a small default cache size setting when netCDF library is built [those who built their own netCDF library with a larger cache got much better response], but today I noticed that if I recreate the grid using `--IO_NC4_CHUNK_SIZE=4096` the speed is back to normal. So, I am wondering if our default selection of chunk sizes (which seems to be in the 128-256 range) is just way too small for at least large grids like this one. Perhaps the _gmtnc_set_optimal_chunksize_ function needs some more scrutiny. Not sure if @ldldr is listening but would like his opinion.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.