[FEATURE REQUEST]: Scalar Expression indices

Summary

This is a feature request for adding support for scalar expression indices (think CREATE INDEX ON t (CAST(col AS text)) / lower(col)) based on a DataFusion expression. Instead of indexing the name column, a user could index lower(name) or concat(first, ' ', last), and filters that contain the same sub-expression would be served by the index.

The expression should be supplied either as a DataFusion Expr or a SQL string. (I would suggest adding the SQL string in a v2).

Design

I had a brief look into this and think the following things need to be implemented.

Storage of the expression itself

Currently, IndexMetadata only stores the columns to build the index. We need to expand this with a new optional field that stores the expression and possibly resulting datatypes.
This requires a change to the proto file and is the major reason why I created this ticket first. I am not sure what the best way would be to land this in multiple PRs. I assume this needs a PMC vote first.

Possible other additions at this stage:

• Store a fingerprint to reduce subtree lookup Expr::eq time.
• Add human readable form for tooling to avoid booting up a DataFusion session.

Index creation/write

Once the proto changes landed, the write path might be easier to implement. We just need to collect used columns from the expression instead of using the columns listed in the index manifest and then apply the expression plan to the training stream.

create_index(expr):   
  1. encode      Substrait-encode the DataFusion Expr against the dataset schema.
  2. resolve     Walk the Expr; collect every Column ref → list of input field ids. Or reuse `fields` and error out if the `Expr` uses a field not part of `fields`.
  3. train       Scan the input columns into RecordBatches, evaluate the Expr per batch
                 to materialize a one-column "indexed values" stream, hand it to the
                 existing scalar-index trainer (btree/bitmap/inverted/…).
  4. record      Persist IndexMetadata with `fields = [input ids]` AND a new optional
                 `IndexedExpression { substrait_expr, output_arrow_type }`.

Query

The query side is harder as we need to change the planner. We might want expression normalization as well, but that can also be implemented in a v2 (similar to https://github.com/lance-format/lance/issues/1549) I think.

query(filter):
  1. for each indexable subtree of `filter`, look up
       (a) by column name (existing path), then
       (b) by canonical-form Expr equality against expression indexes.
  2. matched subtrees → ScalarIndexExpr; unmatched subtrees → refine step.

Plan

1. Create proto change PMV vote
   I assume I have to create a PMC vote issue for the proto changes. What exactly is needed for this?
2. Land proto changes
3. Add index create/write
4. Add query support
5. Add normalization and SQL string support (wondering if SQL string support should live in lancedb instead)

Start by reading the proto definition for IndexMetadata and the existing scalar-index write and query-planning paths. Trace how indexed columns are stored, trained, and matched against filters. Done means a reviewed design and implementation covering expression metadata, index creation, and query use, with tests for supported DataFusion expressions.