incorporate parquet files?

Has there been any discussion of using parquet files at some level of dataverse? (I see it mentioned in only one issue.)

Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk. Apache Parquet is designed to be a common interchange format for both batch and interactive workloads.

I've used them some, and I love how they work well with R, Python, DuckDb, Spark, and others.

Several R programmers (like @kuriwaki) have advocated for rds files over RData files. From my recent experience with parquet files, they have all the advertised advantages of rds files (eg, compression, strong-typing, and factor levels), plus the appeal of interoperability with other platforms.

I haven't thought much beyond this. But when I read about problems with RData files and the messiness of Rserve described by @landreev, I see parquet as a improvement for many reasons --not least is the ability to replace a flaky remote instance with a local parquet library.

cc: @pdurbin

No files, tests, or entry points are named. Start by reviewing issue 6807 and the linked R client discussions about RData and RDS, then define the intended Dataverse scope for Parquet support and its interoperability goals. Done requires an agreed design and implementation scope before coding can begin.