apache / apache/arrow

[Python] Some dataset tests using PyFileSystem are failing on Windows

Open
#31,832 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: test
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

We have some dataset tests that are skipped on Windows, because they are failing with FileNotFound errors.

- https://github.com/apache/arrow/blob/3c3e68c194ca6ac07086ddc1bb44fe153970213e/python/pyarrow/tests/test_dataset.py#L3261-L3264
\*https://github.com/apache/arrow/blob/893faa741f34ee450070503566dafb7291e24d9f/python/pyarrow/tests/test_dataset.py#L3124-L3145 (and see https://github.com/apache/arrow/pull/13033#issuecomment-1116180259 for some analysis)

In the second case, it seems that for some reason, the file paths of the fragments are relative paths to the root of the dataset (while locally for me this gives absolute paths).

**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-16460) / @jorisvandenbossche

**Note**: *This issue was originally created as [ARROW-16460](https://issues.apache.org/jira/browse/ARROW-16460). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the linked sections of python/pyarrow/tests/test_dataset.py, especially the tests around lines 3261-3264 and 3124-3145, and review the analysis in pull request 13033. Run the affected dataset tests on Windows and compare the fragment paths produced by PyFileSystem; done means the tests no longer fail with FileNotFound errors and can be enabled there.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.