apache / apache/beam

Streaming Beam SQL benchmarks on all of our runners

Open
#18,675 0 comments 0 reactions 0 assignees View on GitHub
new feature nexmark P3 sql tests
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Beam has a number of classic streaming SQL benchmarks known as "Nexmark" coded up in both raw Java and also Beam SQL.

So far, expanding functionality has been the focus of Beam SQL so there is little known about performance - we know only that it is a pretty straightforward mapping from SQL to Beam that should work OK a lot of the time. It would be interesting to see where the bottlenecks are when these SQL benchmarks are translated via Beam SQL into a Beam pipeline and then again translated to the native capabilities of e.g. Spark and Flink.

This project will require the ability to read, write, and run Java and SQL.

Imported from Jira [BEAM-3783](https://issues.apache.org/jira/browse/BEAM-3783). Original Jira may contain additional context.
Reported by: kenn.

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing Nexmark implementations in raw Java and Beam SQL, then identify how benchmarks are launched across the available runners. Compare the Beam SQL pipelines with native Spark and Flink execution, and consider the work done when benchmark results and bottlenecks are reported for all runners.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data-engineering, performance, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.