Store decoder logic with the data

It has become somewhat trendy in research to store the decoder implementation alongside the data. See 1 and 2.

The approach provided by the papers is to store the decoders as WASM inside the file itself. This is somewhat undesirable as it requires many duplicate copies of the same decoder implementation and WASM still doesn't quite have the same performance as native. A more problematic issue is that the WASM APIs in the paper typically don't include routines for scheduling and so it is limited to compression and not structural encoding.

However, since we are a table format, a much simpler solution exists, plugins. We can create a stable C ABI for a file reader (it's basically a small addition to the existing Arrow C data interface) and store file decoders in the table as plugins. File decoder plugins would be associated with fragments in the metadata and fragments can share a decoder (e.g. one decoder impl per version, not one per file). Since it is a stable C ABI we don't mandate the use of WASM and dylib (e.g. .so, .dll) could be used if the read hardware is consistent. On the other hand, there is no particular reason we can't use WASM. The current decoder libs are setup so that I/O scheduling is an external process and so they should be easily compiled to wasm. Whatever solution we come up with should tolerate both dylib and wasm plugins.

This would make it possible for older readers to read newer files. That being said, I'm not sure if this is actually a priority / need for anyone or just an interesting research curiosity. I see https://github.com/lancedb/lance/issues/4870 as considerably more important.

The issue names no files, tests, or entry points. First determine whether embedded decoder plugins are a priority, then scope the stable C ABI, fragment metadata association, and support for both dylib and WASM implementations; the work is done when the project has an agreed design and implementation path.