possible use case for xmitgcm

We have an obscure package called xmitgcm which provides read-only access to the binary files generated by the mitgcm ocean model in xarray format. The raw data is usually all stored in a single directory, and is a combination of binary data files (possibly large) and text metadata files (always very small).

Currently, the package works only with files stored on disk. Interaction with the filesystem is basically limited to the following operations:

• os.listdir, to get the files in a particular directory, plus pattern matching (we currently use glob, but it's not necessary)
• open to read and parse metadata stored in text files
• os.path.getsize to examine the size of binary objects
• reading of binary data, as follows:
  • file.seek to point to a specific byte range within the file
  • np.fromfile to load binary data (optionally wrapped in a dask delayed call to make reading lazy). Sometimes invoked with the count argument to read only a specific byte range

We would like to refactor this package to work use a more generic idea of filesystems, with the goal of being able to upload the data "as is" directly into cloud storage buckets and have everything work the same.

Based on my limited understanding of cloud storage, this should be doable. The directory listing and text file reading is trivial. The reading of contiguous byte ranges from binary files should also be possible with range requests.

So the only question is, how do we implement this? We could create our own filesystem abstraction and then implement an on-disk class, an s3 class, an gcs class, etc. But I feel that this would be a waste of time, since these things are already done by s3fs, gcsfs, etc.

So I am here seeking a recommendation about the best way to approach this problem.

cc @raphaeldussin

The issue names no repository files or tests. Start by tracing the filesystem operations listed in the issue and reviewing how fsspec-compatible backends such as s3fs and gcsfs address them. Done means an agreed approach for supporting local and cloud filesystems without duplicating existing abstractions.