lance-format / lance-format/lance

lance table v2.2 is 1.7x larger on s3 than Iceberg Parquet

Open
#7,140 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Hi team, I have table.sanitized.sql table schema in Iceberg stored as parquet and when replicating this into lance table using spark and lance storage format 2.2 with blob v2, we noticed storage size is 1.7x larger. Could you help take a look? Please let me know other info you need, thanks!

Additional information:

  • Spark: OSS 3.5.6
  • Iceberg version: OSS 1.5.2
  • Lance-Spark: OSS latest
  • Iceberg table size:: ~66TiB, lance table ~115TiB
  • blob data size ~90KB average

Lance table creation query

set spark.driver.memory=128g;
set spark.executor.memory=32g;
set spark.executor.memoryOverhead=16g;
set spark.dynamicAllocation.maxExecutors=5000;
set spark.driver.maxResultSize=32g;
set spark.executorEnv.LANCE_INITIAL_UPLOAD_SIZE=5368709120;
SET spark.sql.adaptive.coalescePartitions.enabled=false;
SET spark.sql.shuffle.partitions=50000;

DROP TABLE IF EXISTS pyang_lance_eval_7.lance_multimodal_iceberg_full_blob_v2;

CREATE TABLE IF NOT EXISTS pyang_lance_eval_7.lance_multimodal_iceberg_full_blob_v2
USING lance
TBLPROPERTIES (
  -- 'image_bytes.lance.col1987' = 'blob',
  'col1987.lance.encoding' = 'blob',
  'file_format_version'        = '2.2'
)
AS
SELECT /*+ REPARTITION(50000) */ *
FROM spark_catalog.pyang.lance_multimodal_iceberg_full_iceberg;

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached table.sanitized.sql schema and the CREATE TABLE query, then review the stated Spark, Iceberg, Lance-Spark, and format-version settings. Reproduce or compare storage accounting for the Iceberg and Lance tables, including blob data, and document the cause of the size difference or the configuration change needed to resolve it.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, spark
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.