posit-dev / posit-dev/pointblank

Step reports through `get_step_report()` error in PySpark

Open
#122 0 comments 0 reactions 1 assignee View on GitHub

@rich-iannone is already working on this.

Since Mar 27, 2025.

Difficulty: [2] Intermediate Effort: [2] Medium Priority: [3] High Type: ☹︎ Bug
Dominant language
Python
Stars
490
Forks
32
Avg merge
14h 24m
Merged PRs (30d)
2

Description

A row-based step row (based on a col_vals_*() validation step) currently fails in PySpark. The stack trace I received in recent testing (not well-formatted):

/databricks/spark/python/pyspark/sql/pandas/functions.py:443: UserWarning: In Python 3.6+ and Spark 3.0+, it is preferred to specify type hi nts for pandas UDF instead of specifying pandas UDF type which will be deprecated in the future releases. See SPARK-28264 for more details.
warnings.warn (
/databricks/spark/python/pyspark/sql/pandas/functions.py:443: UserWarning: In Python 3.6+ and Spark 3.0+, it is preferred to specify type hi nts for pandas UDF instead of specifying pandas UDF type which will be deprecated in the future releases. See SPARK-28264 for more details.
warnings. warn (
/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.12/site-packages/pointblank/validate.py:7375: SyntaxWarning: invalid escape
sequence 'Id'
› ValueError: The input table type supplied in 'data=' is not supported.
File <command-4199509693953478>, line 1
- 1 validation.get_step_report (i=1)
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.12/site-packages/pointblank/validate.py:1503, in get_row_count(data)
return data. shape [0]
1500
1502
1>
1503
else:
raise ValueError ("The input table type supplied in 'data=' is not supported.")

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.