apache / apache/wayang

Nonuniform conventions of providing benchmark/demo/example of running Wayang with different platforms

Closed
#796 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
274
Forks
140
Avg merge
5d 16h
Merged PRs (30d)
4

Description

## Problem
In recent PRs #772, #773, #774 we added three new platforms to be supported by Apache Wayang, which is awesome!
However, in doing so, we broke some of the conventions for structuring our code.
We now have multiple ways of providing demo-like examples of how users could run code with specific platforms.
I think this is problematic as it makes it ambiguous for contributors that want to add new code to our project.

## Files
The following files were added in non-conventional manner:
- [Presto-setup](https://github.com/apache/wayang/tree/main/presto-setup)
- [BigQuery-setup](https://github.com/apache/wayang/tree/main/bigquery-setup)
- [Trino-setup](https://github.com/apache/wayang/tree/main/trino-setup)
- [Trino .sh demo](https://github.com/apache/wayang/blob/main/demo-trino.sh)
- [BigQuery .sh demo](https://github.com/apache/wayang/blob/main/demo-bigquery.sh)
- [OS specific .env_template](https://github.com/apache/wayang/blob/main/env_template_osx.sh)

## Previous convention
Demos or examples were previously found only in [wayang-applications](https://github.com/apache/wayang/tree/main/wayang-applications/src/main/java/org/apache/wayang/applications) or in [wayang-benchmark](https://github.com/apache/wayang/tree/main/wayang-benchmark).

## Docker in Wayangs source
Additionally, the demo examples added in #772, #773 and #774 provide docker-compose.yml files for each of the platforms to be set up. While I agree that this is convenient for anyone willing to just execute the demo all batteries included, I do not think that these files should be present in the source of Wayang. An example or demo should be configurable to connect to any third party platform via its API, just by a config like we have in many other examples.
This allows us to not rely on too many third party tools and allows users to plug their environment in more seamlessly.

Contributor guide

Open the contributing guide

Research direction

Compare the listed Presto-, BigQuery-, and Trino-setup files, demo scripts, and env_template_osx.sh with examples under wayang-applications and wayang-benchmark. Establish the existing convention before deciding how these platform examples should be reorganized and configured. Done means platform demos follow one structure and do not require platform-specific Docker Compose files in the Wayang source.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, java, shell
Domain
developer-experience, devops
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.