[DISCUSS] Separate stable API into separate crates

### Is your feature request related to a problem or challenge?

Currently everything in DataFusion is released together which makes the upgrade path painful and causes dependency conflicts if you have a project which depends on multiple 3rd party dependencies which themselves depend on DataFusion.

For example, if I am building a query engine which should support multiple different table formats and want to use 3rd party dependencies which expose DataFusion `TableProvider`s (e.g https://github.com/apache/iceberg-rust and https://github.com/delta-io/delta-rs), but also implement the core execution engine on top of DataFusion then I need to ensure all dependencies are in sync in terms of their DF dependency.

### Describe the solution you'd like

Separate the stable API for internal implementation as separate crates. For instance instead of having just a `datafusion-catalog` crate with both the stable API and implementations we can have two crate:
1. `datafusion-catalog-api`
2. `datafusion-catalog` (which depends on `datafusion-catalog-api`

The `-api` crate would contain only trait definitions like:
`TableProvider`
`CatalogProvider`
etc.

The `-api` should (theoretically) evolve much more slowly than the implementation so 3rd party crates that want to expose a `TableProvider` can depend only on `datafusion-catalog-api` and I can consume that dependency while using any version of `datafusion-catalog` I want (provided it uses the same stable API).

Obviously there are quite a few details to work out about which APIs to "stabilize" in this way, which I will hand-wave past :)

### Describe alternatives you've considered

A lot of alternative suggestions to the exact same issue have been proposed in https://github.com/apache/datafusion/issues/16622

### Additional context

_No response_

Start by reading the alternatives and discussion in issue #16622, then inspect the existing datafusion-catalog crate and its TableProvider and CatalogProvider traits. The issue leaves the APIs to stabilize and the crate boundaries unresolved; done would require an agreed design for the API and implementation crates, including their compatibility expectations.

Written by the indexing model from the issue text.