Improve spark data source for lance

Here I will submit a series of features to improve the spark connector more effectively.

Reader

• Support pushdown for more filters for spark connectors.
  • Pushdown IN to lance scan #3314
• Support topN push down for spark connector. #3261
• Support simple aggreator(count) pushdown for spark connector.
• Support runtime filter pushdown for spark connector.
• Support get the statistics(row_number and data size) for spark optimizer. #3304
• Use the lance index to query data for the spark connector.
• Support pushdown the fragment ids into the lance scanner
• And so on

Writer

• Support overwrite operator for spark sink #3313
• Support merge operator to fast add a new column for spark sink.
• Support AQE for partition number choosen.
• And so on

Catalog

• Impl the fs catalog for spark connector.
• Impl the hms catalog for spark connector.
• And so on

Configuration

• Support set options in lance spark datasource api #3366
• Separate the write params and read options for spark connector.
• Refactor the spark connector configuration since some configurations should persist in the catalog.

This is a broad tracking issue covering reader, writer, catalog, and configuration work for the Spark connector. No files, tests, or single entry point are named; choose one unchecked item and first locate the corresponding connector area before defining its completion criteria.