GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

Support dynamic round-robin chunking in configs run --kube-completions

Open
#1,827 0 comments 0 reactions 1 assignee Claimed by @nj1973 View on GitHub
type: feature request
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

### 1. Summary & Context
This PR adds support for **dynamic round-robin chunking** to the `configs run --kube-completions` (`-kc`) command.

Currently, the `-kc` flag is designed for a strict **1-to-1 sequential numeric mapping**:
* It expects files in the `--config-dir` to be named sequentially (e.g., `0000.yaml`, `0001.yaml`, etc.).
* It maps the `CLOUD_RUN_TASK_INDEX` or `JOB_COMPLETION_INDEX` directly to a single file: `f"{job_index:04d}.yaml"`.

While this works perfectly for validations generated via `generate-table-partitions` (which splits a single massive table into numbered chunks), it **does not scale** for database migrations involving hundreds or thousands of different tables where configuration files are named after their schemas/tables (e.g., `hr.employees.yaml`, `sales.orders.yaml`) rather than sequential numbers.

### 2. Proposed enhancement: Dynamic Round-Robin Chunking
Propose extending `-kc` to automatically detect the **total task count** via standard environment variables (`CLOUD_RUN_TASK_COUNT` or `JOB_COMPLETION_COUNT`).

* **If a total task count is detected**:
1. DVT lists all validation files in the `--config-dir` (supporting GCS and local paths) and sorts them alphabetically.
2. It distributes the files round-robin to the current task using: `my_files = all_files[task_index::task_count]`.
3. It executes the assigned files sequentially within the warm container.
* **If no task count is detected (Legacy Fallback)**:
* DVT falls back to the legacy 1-to-1 sequential numeric naming (e.g., `0000.yaml`). This ensures equivalent behaviour when task count equals file count.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.