Add partition-level pruning for serial vector search

Parent: #6820
Linear: https://linear.app/lancedb/issue/OSS-927/add-partition-level-pruning-for-serial-vector-search
Linear parent: https://linear.app/lancedb/issue/OSS-922/vector-search-performance-optimization-roadmap

Goal

Use serial partition processing in the vector search path to prune entire partitions when they cannot improve the current top-k frontier.

Scope

• Define a valid partition lower-bound strategy for the relevant index types / distance metrics.
• Maintain a global top-k threshold as partitions are searched serially.
• Skip partitions whose lower bound cannot beat the current threshold.
• Preserve recall/correctness guarantees implied by the index and query parameters.
• Make pruning decisions observable enough for benchmarking and debugging.

Acceptance criteria

• Tests cover pruning and no-pruning cases, including edge cases where lower bounds tie the current threshold.
• Recall is unchanged versus the non-pruning path for the same query parameters.
• Benchmarks report partitions visited, partitions pruned, latency, and recall.
• The implementation is gated or disabled when the lower-bound assumptions are not valid for the metric/index type.