lance-format / lance-format/lance
lance table v2.2 is 1.7x larger on s3 than Iceberg Parquet
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Hi team, I have table.sanitized.sql table schema in Iceberg stored as parquet and when replicating this into lance table using spark and lance storage format 2.2 with blob v2, we noticed storage size is 1.7x larger. Could you help take a look? Please let me know other info you need, thanks!
Additional information:
- Spark: OSS 3.5.6
- Iceberg version: OSS 1.5.2
- Lance-Spark: OSS latest
- Iceberg table size:: ~66TiB, lance table ~115TiB
- blob data size ~90KB average
Lance table creation query
set spark.driver.memory=128g;
set spark.executor.memory=32g;
set spark.executor.memoryOverhead=16g;
set spark.dynamicAllocation.maxExecutors=5000;
set spark.driver.maxResultSize=32g;
set spark.executorEnv.LANCE_INITIAL_UPLOAD_SIZE=5368709120;
SET spark.sql.adaptive.coalescePartitions.enabled=false;
SET spark.sql.shuffle.partitions=50000;
DROP TABLE IF EXISTS pyang_lance_eval_7.lance_multimodal_iceberg_full_blob_v2;
CREATE TABLE IF NOT EXISTS pyang_lance_eval_7.lance_multimodal_iceberg_full_blob_v2
USING lance
TBLPROPERTIES (
-- 'image_bytes.lance.col1987' = 'blob',
'col1987.lance.encoding' = 'blob',
'file_format_version' = '2.2'
)
AS
SELECT /*+ REPARTITION(50000) */ *
FROM spark_catalog.pyang.lance_multimodal_iceberg_full_iceberg;
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the attached table.sanitized.sql schema and the CREATE TABLE query, then review the stated Spark, Iceberg, Lance-Spark, and format-version settings. Reproduce or compare storage accounting for the Iceberg and Lance tables, including blob data, and document the cause of the size difference or the configuration change needed to resolve it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, spark
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100