apache / apache/amoro

Clarification Request: Benchmark Details for Amoro vs. Spark Compaction Blog

Open
#4,173 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
395
Avg merge
4d 10h
Merged PRs (30d)
33

Description

I recently read the Amoro 10x Efficient blog on Medium:
https://medium.com/@jinsong.zhou1990/10x-efficiency-boost-compared-to-spark-rewritefiles-procedure-how-apache-amoro-efficiently-7e7a993950d7

The performance improvements demonstrated are highly impressive. However, I noticed a few missing nuances in the benchmark setup. We would love to get more details on the following points to better understand the comparison.

1. The CREATE TABLE statement shows the schema, but it does not specify the primary key. Was it a single primary key or a composite key?
2. What type of CDC operations were applied during the benchmark? Was it update operation or just inserts happening or only deletes happening?
3. Since only the schema is provided, could you share what dataset was used for this comparison?
4. What were the specific Amoro optimizer parameters and configurations set during the benchmark?
5. The blog does not mention partitioning. Was the table partitioned, and if so, what was the partition spec?
6. Which exact version of Amoro was used for this benchmarking?
7. What was the level of task parallelism configured for both Amoro and Spark during this test?
8. Before running the compaction, was the data sorted in the data files based on the primary key column?
9. Which specific column or columns were used as the equality identifiers for the equality delete files generated during the CDC workload?
10. Were the data files sorted based on the equality delete column?
11. Amount of data that used per CDC event? For instance 100k rows updated. If you could inform what kind of data was being changed during the CDC event it would be really helpful.

Any additional context you can provide regarding the test environment and workload would be incredibly helpful for the community. Thank you for the great article and your hard work on the project!

Contributor guide

Open the contributing guide

Research direction

Start with the linked Medium article and compare its benchmark description with the available Amoro and Spark test context. Done means publishing a maintainer-confirmed response covering the schema, CDC workload, dataset, configuration, versions, partitioning, sorting, parallelism, and data volume questions.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.