Cleanup Bigquery dataset after Python Mobile Game validation
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
run_rc_validation.sh validates Python GameStats and Leaderboard in streaming. Before each pipeline starts, a unique bq dataset is created but never cleanup after pipeline is done or script is interrupted.
Imported from Jira [BEAM-8349](https://issues.apache.org/jira/browse/BEAM-8349). Original Jira may contain additional context.
Reported by: markflyhigh.
Subtask of issue #19755
Contributor guide
Research direction
Start by reading run_rc_validation.sh and the Python GameStats and Leaderboard validation flow. Trace where each unique BigQuery dataset is created and how the pipeline exits or is interrupted. Done means datasets are cleaned up after successful runs and when the script is interrupted; check the original BEAM-8349 Jira issue for additional context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100