IPF replacement format

In GitLab by @Huite on Mar 8, 2022, 13:45

Ideally we could replace the IPF format. Reading is quite slow and there are few guarantees with regards to consistencies (delimiter, column names, etc.).

NetCDF is one option, but is difficult to implement with regards to QGIS. QGIS integration is really a key point, as we'd like to use this format to easily visualize. This rather limits options: if QGIS interprets it as raster data, it's fundamentally unsuitable. This basically excludes netCDF, since within QGIS we'd rely on GDAL to read netCDF.

I think a reasonable candidate might be two Tab-Separated-Values files (TSV):

• One with the static data (location, surface, whatever)
• One with ALL timeseries: this guarantees consistency atleast for the column names

Together in a single ZIP file. The logic for this is extremely easy to write, requires zipfile in Python standard library and pandas read csv.

This is also easily workable from QGIS (zipfile, read delimited file).

This might also work from a geopackage (but always requires something like geopandas to write from Python!).
But the problem is that within QGIS, you cannot use the relatively efficient pandas read functions; one has to use the slow Python QGIS API to fetch row by row.

The issue names no repository files, tests, or entry points. Start by locating the existing IPF reading and writing logic, then compare the proposed two-TSV-in-a-ZIP format with NetCDF and GeoPackage for consistency, performance, and QGIS compatibility. Done requires an agreed replacement format and an implementation plan.