GoogleCloudPlatform / GoogleCloudPlatform/professional-services-data-validator

generate-table-partitions make error

Open
#1,705 1 comment 0 reactions 0 assignees View on GitHub
type: bug
Dominant language
Python
Stars
524
Forks
171
Avg merge
5d 15h
Merged PRs (30d)
4

Description

**Describe the bug**

The YAML file generated using the generate-table-partitions method contains an SQL statement with the :param_1 parameter. When subsequently running config run, the following error occurs:
parse error: Unfinished string at EOF at line 1, column 248180736

**What version of DVT are you using?**
DVT: 8.5.2, python 3.11

**What type of connections are you using for source and target?**
Ex: Mysql 8.x for source, PG 9.x for target

**Steps to reproduce the behavior**
1. Add 'data-validation generate-table-partitions -sc mysql -tc dws \
-pk "l_orderkey,l_partkey,l_suppkey,l_linenumber" \
--tables-list "test_db.lineitem=public.lineitem" \
--hash "l_shipinstruct" \
-cdir partions \
--partition-num 10 \
--format json'
2. Execute 'data-validation -v -ll DEBUG configs run -c partions/test_db.lineitem/0000.yaml | jq '.''
3. Scroll down to '03/19/2026 07:27:29 PM-INFO: -- ** Source Query ** --
03/19/2026 07:27:29 PM-INFO: SELECT t0.hash__all, t0.l_orderkey, t0.l_partkey, t0.l_suppkey, t0.l_linenumber
FROM (SELECT t1.l_orderkey AS l_orderkey, t1.l_partkey AS l_partkey, t1.l_suppkey AS l_suppkey, t1.l_linenumber AS l_linenumber, t1.l_quantity AS l_quantity, t1.l_extendedprice AS l_extendedprice, t1.l_discount AS l_discount, t1.l_tax AS l_tax, t1.l_returnflag AS l_returnflag, t1.l_linestatus AS l_linestatus, t1.l_shipdate AS l_shipdate, t1.l_commitdate AS l_commitdate, t1.l_receiptdate AS l_receiptdate, t1.l_shipinstruct AS l_shipinstruct, t1.l_shipmode AS l_shipmode, t1.l_comment AS l_comment, t1.cast__l_shipinstruct AS cast__l_shipinstruct, t1.ifnull__cast__l_shipinstruct AS ifnull__cast__l_shipinstruct, t1.rstrip__ifnull__cast__l_shipinstruct AS rstrip__ifnull__cast__l_shipinstruct, t1.concat__all AS concat__all, sha2(t1.concat__all, '256') AS hash__all
FROM (SELECT t2.l_orderkey AS l_orderkey, t2.l_partkey AS l_partkey, t2.l_suppkey AS l_suppkey, t2.l_linenumber AS l_linenumber, t2.l_quantity AS l_quantity, t2.l_extendedprice AS l_extendedprice, t2.l_discount AS l_discount, t2.l_tax AS l_tax, t2.l_returnflag AS l_returnflag, t2.l_linestatus AS l_linestatus, t2.l_shipdate AS l_shipdate, t2.l_commitdate AS l_commitdate, t2.l_receiptdate AS l_receiptdate, t2.l_shipinstruct AS l_shipinstruct, t2.l_shipmode AS l_shipmode, t2.l_comment AS l_comment, t2.cast__l_shipinstruct AS cast__l_shipinstruct, t2.ifnull__cast__l_shipinstruct AS ifnull__cast__l_shipinstruct, t2.rstrip__ifnull__cast__l_shipinstruct AS rstrip__ifnull__cast__l_shipinstruct, concat_ws(:param_1, t2.rstrip__ifnull__cast__l_shipinstruct) AS concat__all
'
4. 5. See error '03/19/2026 07:28:03 PM-INFO: {"run_id": "8103a851-a491-458f-b6fa-77379a8591e1", "start_time": "2026-03-19T11:27:29.642633+00:00", "end_time": "2026-03-19T11:27:43.132156+00:00", "total_source_rows": 1199799, "total_target_rows": 1199799, "total_rows_validated": 1199799, "total_rows_success_validation_status": 1199799, "total_rows_fail_validation_status": 0, "failed_rows_present_in_source_not_in_target": 0, "failed_rows_present_in_target_not_in_source": 0, "failed_rows_present_in_both_source_and_target": 0}
03/19/2026 07:28:03 PM-DEBUG: Generate report elapsed: 22.75s
parse error: Unfinished string at EOF at line 1, column 248180736'
**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**How are you executing DVT?**
Ex: CLI commands on local computer or VM, within a DAG, Cloud Run job, etc

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.