posit-dev / posit-dev/pointblank
Step reports through `get_step_report()` error in PySpark
Open
@rich-iannone is already working on this.
Since Mar 27, 2025.
Difficulty: [2] Intermediate
Effort: [2] Medium
Priority: [3] High
Type: ☹︎ Bug
- Dominant language
- Python
- Stars
- 490
- Forks
- 32
- Avg merge
- 14h 24m
- Merged PRs (30d)
- 2
Description
A row-based step row (based on a col_vals_*() validation step) currently fails in PySpark. The stack trace I received in recent testing (not well-formatted):
/databricks/spark/python/pyspark/sql/pandas/functions.py:443: UserWarning: In Python 3.6+ and Spark 3.0+, it is preferred to specify type hi nts for pandas UDF instead of specifying pandas UDF type which will be deprecated in the future releases. See SPARK-28264 for more details.
warnings.warn (
/databricks/spark/python/pyspark/sql/pandas/functions.py:443: UserWarning: In Python 3.6+ and Spark 3.0+, it is preferred to specify type hi nts for pandas UDF instead of specifying pandas UDF type which will be deprecated in the future releases. See SPARK-28264 for more details.
warnings. warn (
/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.12/site-packages/pointblank/validate.py:7375: SyntaxWarning: invalid escape
sequence 'Id'
› ValueError: The input table type supplied in 'data=' is not supported.
File <command-4199509693953478>, line 1
- 1 validation.get_step_report (i=1)
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.12/site-packages/pointblank/validate.py:1503, in get_row_count(data)
return data. shape [0]
1500
1502
1>
1503
else:
raise ValueError ("The input table type supplied in 'data=' is not supported.")
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.