[Python] Some dataset tests using PyFileSystem are failing on Windows
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
We have some dataset tests that are skipped on Windows, because they are failing with FileNotFound errors.
- https://github.com/apache/arrow/blob/3c3e68c194ca6ac07086ddc1bb44fe153970213e/python/pyarrow/tests/test_dataset.py#L3261-L3264
\*https://github.com/apache/arrow/blob/893faa741f34ee450070503566dafb7291e24d9f/python/pyarrow/tests/test_dataset.py#L3124-L3145 (and see https://github.com/apache/arrow/pull/13033#issuecomment-1116180259 for some analysis)
In the second case, it seems that for some reason, the file paths of the fragments are relative paths to the root of the dataset (while locally for me this gives absolute paths).
**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-16460) / @jorisvandenbossche
**Note**: *This issue was originally created as [ARROW-16460](https://issues.apache.org/jira/browse/ARROW-16460). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with the linked sections of python/pyarrow/tests/test_dataset.py, especially the tests around lines 3261-3264 and 3124-3145, and review the analysis in pull request 13033. Run the affected dataset tests on Windows and compare the fragment paths produced by PyFileSystem; done means the tests no longer fail with FileNotFound errors and can be enabled there.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100