Change `serde` dependency to `serde_core` where applicable

### Is your feature request related to a problem or challenge?

Currently some datafusion crates depend on `serde`, which may take a long time to compile if -- due to feature unification -- datafusion users opted into the `derive` features. See https://github.com/serde-rs/serde/pull/2608 . However we often don't need `derive` and hence we could depend on the actual core part of serde called `serde_derive`.

### Describe the solution you'd like

Check our dependencies and where we don't use `derive`, change the dependency from `serde` to `serde_derive`.

### Describe alternatives you've considered

\-

### Additional context

- https://github.com/serde-rs/serde/pull/2608
- https://github.com/apache/arrow-rs/issues/8451