GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator
Support dynamic round-robin chunking in configs run --kube-completions
- Dominant language
- Python
- Stars
- 524
- Forks
- 171
- Avg merge
- 5d 15h
- Merged PRs (30d)
- 4
Description
### 1. Summary & Context
This PR adds support for **dynamic round-robin chunking** to the `configs run --kube-completions` (`-kc`) command.
Currently, the `-kc` flag is designed for a strict **1-to-1 sequential numeric mapping**:
* It expects files in the `--config-dir` to be named sequentially (e.g., `0000.yaml`, `0001.yaml`, etc.).
* It maps the `CLOUD_RUN_TASK_INDEX` or `JOB_COMPLETION_INDEX` directly to a single file: `f"{job_index:04d}.yaml"`.
While this works perfectly for validations generated via `generate-table-partitions` (which splits a single massive table into numbered chunks), it **does not scale** for database migrations involving hundreds or thousands of different tables where configuration files are named after their schemas/tables (e.g., `hr.employees.yaml`, `sales.orders.yaml`) rather than sequential numbers.
### 2. Proposed enhancement: Dynamic Round-Robin Chunking
Propose extending `-kc` to automatically detect the **total task count** via standard environment variables (`CLOUD_RUN_TASK_COUNT` or `JOB_COMPLETION_COUNT`).
* **If a total task count is detected**:
1. DVT lists all validation files in the `--config-dir` (supporting GCS and local paths) and sorts them alphabetically.
2. It distributes the files round-robin to the current task using: `my_files = all_files[task_index::task_count]`.
3. It executes the assigned files sequentially within the warm container.
* **If no task count is detected (Legacy Fallback)**:
* DVT falls back to the legacy 1-to-1 sequential numeric naming (e.g., `0000.yaml`). This ensures equivalent behaviour when task count equals file count.
Contributor guide
Assessment
This issue has not been assessed yet.