[PYTHON][PYSPARK][FUNCTIONS] pyspark.sql.functions.try_to_date is missing in PySpark 4.0.0-4.0.4 but the docs say it became available in 4.0.0
- Dominant language
- Scala
- Stars
- 44k
- Forks
- 29.4k
- PR merge metrics
- No merged PRs in 30d
Description
## The issue
`pyspark.sql.functions.try_to_date` does not appear to exist in PySpark 4.0.*.
I cannot import or access it from `pyspark.sql.functions`, and I also cannot see it in the Spark 4.0 branch source:
https://github.com/apache/spark/blob/branch-4.0/python/pyspark/sql/functions/builtin.py
By contrast, `pyspark.sql.functions.try_to_date` does appear to exist in PySpark 4.1.*:
https://github.com/apache/spark/blob/branch-4.1/python/pyspark/sql/functions/builtin.py#L11475
https://github.com/apache/spark/blob/fb14e6cee794191b8a27f3acc51e695492edeaa6/python/pyspark/sql/functions/builtin.py#L11475
The current docs say that `pyspark.sql.functions.try_to_date` is:
> New in version 4.0.0
https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.try_to_date.html
That appears misleading.
Presumably this is either:
1. correct documentation, but a missing export / packaging issue in 4.0.0-4.0.4, or
2. incorrect documentation, and `try_to_date` was actually introduced in 4.1.x
## Impact
PySpark 4.0.* enables spark.sql.ansi.enabled by default meaning there's stricter parsing/cast behaviour than in PySpark 3.* , but try_to_date is not available meaning the expected safe null-tolerant alternative is missing. This creates an awkward gap for (try_)to_date users in PySpark version 4.0.*.
## Reproduction
In an environment with PySpark 4.0.1, 4.0.2, 4.0.3, or 4.0.4 installed:
```python
import pyspark
from pyspark.sql import functions as F
print(pyspark.__version__)
print(hasattr(F, "try_to_date"))
print(hasattr(F, "try_to_timestamp"))
```
Output:
```
4.0.4
False
True
```
Trying to use `F.try_to_date(...)` fails with:
> AttributeError: module 'pyspark.sql.functions' has no attribute 'try_to_date'
## Expected behavior
I think that one of the following should be true:
1. if try_to_date is intended to be available in 4.0.*, it should be present in pyspark.sql.functions in those releases
2. if it was only introduced in 4.1.*, the documentation should say that instead of claiming 'New in version 4.0.0'
Contributor guide
Research direction
Compare the branch-4.0 and branch-4.1 versions of python/pyspark/sql/functions/builtin.py, then check the API documentation for try_to_date. Run the supplied hasattr reproduction against a PySpark 4.0.x environment. Done means either try_to_date is available in the affected releases or its documented introduction version matches reality.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100