Adding alternatives to the exchange of record-oriented json messages

As great and universal as json is for exchanging messages, the fact that Singer tap -> target communication requires a record-oriented json format is a big drawback (at least for me), as the unnecessary serialisation/deserialisation overhead becomes a real pain when processing (analytical) bulk data.

I was wondering how much room/importance the community sees in extending the spec in this area.

My first thought would be to add the option to use Apache Arrow Inter Process Communication (IPC). For those unfamiliar with Arrow: Arrow is a standardised columnar memory specification, and IPC is a way of transferring arrow record batches without the need for serialisation/deserialisation. As many data storage systems are adopting Arrow Flight, the will be a lot of value in data pipelines that use Arrow as the shared data layout in every step from extraction to loading.

Let me know if this is of interest, or if more information is needed, and I will add more detail.

Start by reviewing the Singer tap-to-target record-oriented JSON communication described in the issue, then read the linked Apache Arrow IPC and Arrow Flight references. The issue does not name files or tests; the work would first require defining whether an alternative transport belongs in the Singer spec and what requirements it must meet for bulk analytical data.