dataset.to_table().to_pandas() raise error `Io error: Execution error: LanceError(Arrow): Invalid argument error`

example code

import lance
import pyarrow as pa
import numpy as np

train_data_size = 100000
test_data_size = 100
vector_dimension = 768
cluster_number = 10
cluster_size = train_data_size // cluster_number

train_data = []
for i in range(cluster_number):
    cluster_center = np.random.rand(vector_dimension)
    cluster_data = np.random.randn(cluster_size, vector_dimension) + cluster_center
    train_data.append(cluster_data)
# save train data to parquet file
train_data = np.concatenate(train_data)
train_data = np.array(train_data, dtype=np.float32)

table = pa.table({
   "id": pa.array([x for x in range(train_data_size)]),
   "embedding": pa.array(train_data.tolist())
})
dataset = lance.write_dataset(table, "embeddings")
df = dataset.to_table().to_pandas()
print(df.head())

error

Traceback (most recent call last):
 xxx
  File "pyarrow/ipc.pxi", line 750, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: C Data interface error: Io error: Execution error: LanceError(Arrow): Invalid argument error: Last offset 6290688 of List(Field { name: "item", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) is larger than values length 8192

Run the provided Python reproduction with the listed dependencies, then trace dataset.to_table().to_pandas() through the Arrow C Data interface shown in the traceback. Done means the example completes and prints the DataFrame without the List offset/values-length ArrowInvalid error.