Multi-column types not correctly handled in SortByComponent

SortByComponent::FastField { field_name, .. } => {
    let sort_column_opt: Option<(Column<u64>, ColumnType)> =
        segment_reader.fast_fields().u64_lenient(field_name)?; // <-- returns first column it encounters
    let (sort_column, column_type) = sort_column_opt.unwrap_or_else(|| {
        (
            Column::build_empty_column(segment_reader.max_doc()),
            ColumnType::U64,
        )
    });
    let sort_field_type = SortFieldType::try_from(column_type)?;
    Ok(SortingFieldExtractorComponent::FastField {
        sort_column,
        sort_field_type,
    })
}

u64_lenient returns the first column it encounters, but there may be potential multiple types. The order of the columns is defined by their lexicographic ordering of their serialized version in tantivy's columnar storage.

This may lead to unexpected behavior on JSON fields with mixed types. Especially behavior may appear random, depending on what data ends up in which Split.

With search_after we probably want to try to match the colum_type to the parameter passed. Maybe have some priority list of accepted types.

Current ColumnType Order:

#[repr(u8)]
pub enum ColumnType {
    I64 = 0u8,
    U64 = 1u8,
    F64 = 2u8,
    Bytes = 3u8,
    Str = 4u8,
    Bool = 5u8,
    IpAddr = 6u8,
    DateTime = 7u8,
}

Note

This only applies to JSON fields, as only they can have mixed types currently. This includes anything in _dynamic.

Start at the SortByComponent::FastField path and trace how u64_lenient selects a column and how ColumnType becomes SortFieldType. Reproduce the behavior with mixed-type JSON or _dynamic fields, then define and test deterministic type selection that also works with search_after.