[CI][Python] test_arrow_cogrouped_map and test_arrow_grouped_map from pyspark fail on integration job
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
As seen here:
https://github.com/apache/arrow/pull/44981#issuecomment-2529087381
When I tried to run "pyspark.sql.tests.arrow.test_arrow_grouped_map" and "pyspark.sql.tests.arrow.test_arrow_cogrouped_map" they fail due to missing pandas:
```
Traceback (most recent call last):
File "/spark/python/pyspark/sql/tests/arrow/test_arrow_grouped_map.py", line 264, in test_self_join
df2 = df.groupby("k").applyInArrow(arrow_func, schema="x long, y long")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/spark/python/pyspark/sql/pandas/group_ops.py", line 809, in applyInArrow
udf = pandas_udf(
^^^^^^^^^^^
File "/spark/python/pyspark/sql/pandas/functions.py", line 372, in pandas_udf
require_minimum_pandas_version()
File "/spark/python/pyspark/sql/pandas/utils.py", line 43, in require_minimum_pandas_version
raise PySparkImportError(
pyspark.errors.exceptions.base.PySparkImportError: [PACKAGE_NOT_INSTALLED] Pandas >= 2.0.0 must be installed; however, it was not found.
```
Those tests were never executed in the past but might be worth to include them on the job as they are arrow related.
### Component(s)
Continuous Integration, Python
Contributor guide
Assessment
This issue has not been assessed yet.