apache / apache/spark

[PYTHON][PYSPARK][FUNCTIONS] pyspark.sql.functions.try_to_date is missing in PySpark 4.0.0-4.0.4 but the docs say it became available in 4.0.0

Open
#56,672 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
44k
Forks
29.4k
PR merge metrics
No merged PRs in 30d

Description

## The issue

`pyspark.sql.functions.try_to_date` does not appear to exist in PySpark 4.0.*.

I cannot import or access it from `pyspark.sql.functions`, and I also cannot see it in the Spark 4.0 branch source:
https://github.com/apache/spark/blob/branch-4.0/python/pyspark/sql/functions/builtin.py

By contrast, `pyspark.sql.functions.try_to_date` does appear to exist in PySpark 4.1.*:
https://github.com/apache/spark/blob/branch-4.1/python/pyspark/sql/functions/builtin.py#L11475
https://github.com/apache/spark/blob/fb14e6cee794191b8a27f3acc51e695492edeaa6/python/pyspark/sql/functions/builtin.py#L11475

The current docs say that `pyspark.sql.functions.try_to_date` is:

> New in version 4.0.0

https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.try_to_date.html

That appears misleading.

Presumably this is either:

1. correct documentation, but a missing export / packaging issue in 4.0.0-4.0.4, or
2. incorrect documentation, and `try_to_date` was actually introduced in 4.1.x

## Impact

PySpark 4.0.* enables spark.sql.ansi.enabled by default meaning there's stricter parsing/cast behaviour than in PySpark 3.* , but try_to_date is not available meaning the expected safe null-tolerant alternative is missing. This creates an awkward gap for (try_)to_date users in PySpark version 4.0.*.

## Reproduction

In an environment with PySpark 4.0.1, 4.0.2, 4.0.3, or 4.0.4 installed:

```python
import pyspark
from pyspark.sql import functions as F

print(pyspark.__version__)
print(hasattr(F, "try_to_date"))
print(hasattr(F, "try_to_timestamp"))
```

Output:
```
4.0.4
False
True
```

Trying to use `F.try_to_date(...)` fails with:
> AttributeError: module 'pyspark.sql.functions' has no attribute 'try_to_date'

## Expected behavior

I think that one of the following should be true:
1. if try_to_date is intended to be available in 4.0.*, it should be present in pyspark.sql.functions in those releases
2. if it was only introduced in 4.1.*, the documentation should say that instead of claiming 'New in version 4.0.0'

Contributor guide

Open the contributing guide

Research direction

Compare the branch-4.0 and branch-4.1 versions of python/pyspark/sql/functions/builtin.py, then check the API documentation for try_to_date. Run the supplied hasattr reproduction against a PySpark 4.0.x environment. Done means either try_to_date is available in the affected releases or its documented introduction version matches reality.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.