potential performance improvements for GRIB files

I've been playing around a bit on reading GRIB files, but quickly became hit by the performance impact of the temporary files being created by kerchunk/grib2.py. Thus I tried to find ways around this. As far as I understood up to now, cfgrib requires access to entire files and also requires some file-API while eccodes is happy with in-memory grib messages as well. So I tried to read in grib files using mostly eccodes and circumventing cfgrib where possible, which is orders of magnitude faster than the current method implemented in kerchunk, but sadly, it doesn't do all the magic cfgrib does in assembling proper datasets in all cases. This lack of generality is the reason why I'm not proposing a PR (yet?), but rather seek for further ideas on that topic:

• Do others work on this as well?
• Do you have ideas on how to do the dataset assembly more generically?

Here's how I'd implement the "decompression", which I belive is relatively generic (but may still be incompatible with what the current kerchunk-grib does):

import eccodes
import numcodecs
from numcodecs.compat import ndarray_copy, ensure_contiguous_ndarray

class RawGribCodec(numcodecs.abc.Codec):
    codec_id = "rawgrib"

    def encode(self, buf):
        return buf

    def decode(self, buf, out=None):
        mid = eccodes.codes_new_from_message(bytes(buf))
        try:
            data = eccodes.codes_get_array(mid, "values")
        finally:
            eccodes.codes_release(mid)

        if hasattr(data, "build_array"):
            data = data.build_array()


        if out is not None:
            return ndarray_copy(data, out)
        else:
            return data

this gist shows how it may be possible to scan GRIB files without the need for temporary files

Start with kerchunk/grib2.py and compare its temporary-file path with the in-memory eccodes approach described in the issue and gist. Read how cfgrib currently assembles datasets, then determine whether a generic assembly strategy can preserve that behavior without temporary files. Done means an agreed implementation direction with benchmarks and a sufficiently general scope for a PR.