lance-format / lance-format/lance

Blob encoding metadata not applied when writing Lance dataset via Ray Data mapfunction

Open
#5,180 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ray
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Steps to reproduce:

import lance
import pyarrow as pa
import ray.data
import lance
import pyarrow as pa
import pandas as pd
from pathlib import Path
import random

schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field("video_name", pa.string()),
    pa.field("video_size", pa.int64()),
    pa.field(
        "video_data",
        pa.large_binary(),
        metadata={"lance-encoding:blob": "true"}
    ),
])

mp4_files_dir="/DATA/workshop/personal/Lance/HDFS"
mp4_dir = Path(mp4_files_dir)
mp4_files = list(mp4_dir.glob("*.mp4"))

if not mp4_files:
    print(f"在目录 {mp4_files_dir} 中未找到 MP4 文件")

print(f"找到 {len(mp4_files)} 个 MP4 文件")

class Video:
    def __init__(self):
        pass

    def __call__(self, row):
        print(f"this row {row}")
        mp4_file = random.choice(mp4_files)
        try:
            with open(mp4_file, 'rb') as f:
                video_bytes = f.read()

            row[f"video_name"] = mp4_file.name
            row[f"video_size"] = mp4_file.stat().st_size
            row[f"video_data"] = video_bytes

            print(f"已加载: {mp4_file.name} ({len(video_bytes)} 字节)")

        except Exception as e:
            print(f"读取文件 {mp4_file} 时出错: {e}")
        return row


LANCE_PATH = "s3://tmp/test1.lance"


def write():

    ray.data.range(100).map(Video).write_lance(
        LANCE_PATH, mode="create", min_rows_per_file=100, storage_options=storage_options, schema=schema)


def read():
    df = lance.dataset(LANCE_PATH, storage_options=storage_options)
    print(
        df.to_table().to_pandas()
    )

print(ray.data.range(100).map(Video).schema())
write()

Expected behavior:

The video_datafield should be stored with blob encoding as specified in the schema metadata.

Actual behavior:

The field is stored as regular binary data without blob encoding.

Show table result

Image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided Ray Data reproduction and compare the schema returned by ray.data.range(100).map(Video).schema() with the schema passed to write_lance. Trace how the video_data field metadata is propagated through write_lance and into lance.dataset, then verify that the stored field retains blob encoding metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.