Add support for Dict(K,V) as scalar index (BTree and Bitmap) on nullable and non-nullable columns

It currently is not possible to create a scalar index for a column of type Dict(u16, Utf8).
On a nullable dictionary column, the BTree index fails to write page_lookup.lance (≥ v2.1 encoder).

The encoder rejects the resulting empty-values dictionary:

create BTree on nullable dict => 
  Err(Arrow { 
    message: "Invalid argument error: Value at position 0 out of bounds: 0 (should be in [0, -1])", 
    location: Location { file: "rust/lance-encoding/src/encodings/logical/primitive.rs", 
    line: 5665, 
    column: 33 
  } })

thread 'dataset::scanner::test::repro_btree_dict_null_crash' (193226154) panicked at rust/lance/src/dataset/scanner.rs:5371:13:

called `Result::unwrap()` on an `Err` value: Arrow { 
  message: "Invalid argument error: Value at position 0 out of bounds: 0 (should be in [0, -1])", 
  location: Location { file: "rust/lance-encoding/src/encodings/logical/primitive.rs", 
  line: 5665, 
  column: 33 
} }

Repro:

#[tokio::test]
async fn repro_btree_dict_null_crash() {
   use arrow_array::{Int16Array, Int16DictionaryArray};
   use lance_index::scalar::BuiltinIndexType;

   // Dictionary(Int16, Utf8) column with a null, small enough for a single
   // btree page.
   let schema = Arc::new(ArrowSchema::new(vec![ArrowField::new(
      "category",
      DataType::Dictionary(Box::new(DataType::Int16), Box::new(DataType::Utf8)),
      true,
   )]));
   let dict = Arc::new(StringArray::from(vec!["a", "b", "c"]));
   let keys = Int16Array::from(vec![Some(0), Some(1), Some(2), None]); // a,b,c,null
   let arr = Int16DictionaryArray::try_new(keys, dict).unwrap();
   let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(arr)]).unwrap();
   let reader = RecordBatchIterator::new(vec![Ok(batch)], schema.clone());
   let mut dataset = Dataset::write(reader, "memory://repro_btree_dict", None)
      .await
      .unwrap();

   let params = ScalarIndexParams::for_builtin(BuiltinIndexType::BTree);
   let res = dataset
      .create_index(&["category"], IndexType::Scalar, None, &params, true)
      .await;
   println!("create BTree on nullable dict => {res:?}");
   res.unwrap();
}

General Issues With Dicts

Dicts for the same column are not guaranteed to be equal over multiple RecordBatches, as far as I understand. For example, if they are received via IPC, where a dict can be replaced completely in a second RecordBatch. See https://arrow.apache.org/docs/format/Columnar.html#dictionary-messages

It therefore makes sense to use the logical values of a dict column in the index. This should at least yield the same storage requirement as a BTree or Bitmap index on a Utf8 column (talking about Dictionary(_, Utf8)).

Start with the repro_btree_dict_null_crash test and the create_index call in the issue, then inspect rust/lance-encoding/src/encodings/logical/primitive.rs around line 5665 and rust/lance/src/dataset/scanner.rs around line 5371. Run the nullable dictionary BTree repro first, then cover nullable and non-nullable Dict(u16, Utf8) cases for BTree and Bitmap. Done means scalar index creation succeeds and uses logical dictionary values.