[EPIC] Support `VARIANT` type for unstructured data

### Is your feature request related to a problem or challenge?

Processing semi-structured data (basically think anything that can be represented in JSON) efficiently is becoming more and more important.

As @wjones127 says in https://github.com/apache/datafusion/issues/10987>

> This would be a high-performance data type for semi-structured data, designed for better OLAP performance than JSON or BSON (discussed in #7845).

While it is certainly possible to implement semi-structured, JSON and even Variant support today using the DataFusion extension apis (e.g. https://github.com/datafusion-contrib/datafusion-functions-json) this ticket tracks adding such support to DataFusion itself

Parquet recently adopted the Variant type : https://github.com/apache/parquet-format/blob/master/VariantEncoding.md

We see adoption of this in other systems as well such as Iceberg and Spark.
- https://github.com/apache/iceberg/issues/10392

I think DataBricks did a good job describing its rationale:
- https://www.databricks.com/blog/introducing-open-variant-data-type-delta-lake-and-apache-spark

> Without Variant, customers had to choose between flexibility and performance. To maintain flexibility, customers would store JSON in single columns as strings. To see better performance, customers would apply strict schematizing approaches with structs, which requires separate processes to maintain and update with schema changes. With Variant, customers can retain flexibility (there's no need to define an explicit schema) and receive vastly improved performance compared to querying the JSON as a string.

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

This will be a big project. Here are some of the related pre-requisites

- [x] https://github.com/apache/arrow-rs/issues/6736
- [x] https://github.com/apache/arrow-rs/issues/8480
- [x] https://github.com/apache/datafusion/issues/14993
- [ ] https://github.com/apache/datafusion/issues/12644

It is not clear to me if variant should be "built in" or if it should be an add on (for example, add a `variant` feature and a `datafusion-variant` crate)

### Additional context

Related tickets
- https://github.com/apache/datafusion/issues/10987
- https://github.com/apache/datafusion/discussions/15264
- https://github.com/apache/datafusion/discussions/9103

Start by reviewing the completed Arrow prerequisites and the open DataFusion prerequisite in #12644, then read related tickets #10987, #14993, and discussion #15264. The epic does not name files or tests; done would require resolving the built-in versus add-on design and delivering VARIANT support throughout DataFusion.

Written by the indexing model from the issue text.