Expose DataFusion extensions (plan nodes, plan optimizer rules)

The code in dataset/scanner.rs has gotten extremely complicated, to a point where it is hard to test. Before we make any improvements, we need to refactor this to be easier to test and extend.

In addition, outside codebases may wish to extend Lance's capabilities by modifying or composing plans. For example, in LanceDB, we'll want to add a separate WAL that needs to be queried during KNN queries and scans.

Tasks

• Create a LogicalPlan for KNN search
• Write an extension for LogicalPlanBuilder and rewrite Scanner::create_plan() in terms of that.
• Create a custom PhysicalPlanner to convert out logical plans into relevant nodes
• Implement a PhysicalOptimizerRule for each of the optimizations we make:
  • Using vector indices (ANN)
  • Using scalar indices (ANN and scan)
• Make sure schema returned from scanner comes from the logical plan.
• metadata is returned sometimes but not all times. Make this behavior more consistent.
• make _rowid and _distance less awkward to deal with -- so that select([]).with_row_id() is just select ("_rowid") We should reserve that column name.
• don't return _distance unless select * or select _distance

Start with rust/lance/src/dataset/scanner.rs and review the linked DataFusion LogicalPlan, LogicalPlanBuilder, PhysicalPlanner, and PhysicalOptimizerRule interfaces. Trace Scanner::create_plan() and map the listed KNN, optimizer, schema, metadata, _rowid, and _distance tasks to the relevant plan boundaries. Done means the requested extensions and consistent scanner behavior are implemented across the checklist.