[C++] Reconsider filesystem equality
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Filesystems support an equality method to compare filesystem instances. The original idea is that all filesystem parameters should be transparent and easily read back, so it should be possible to support equality (similarly, it was envisioned to allow roundtripping filesystems through URIs, though the filesystem-to-URI direction was never implemented).
However, along the way, filesystems like S3 grew increasingly complex and opaque modes of configuration where equality can only be approximated. It can also be costly to compute (for example, `S3Options::Equals` involves fetching the actual secret key and session token, which can take some time: these mere operations consume 5 seconds in the PyArrow test suite).
Right now, filesystem equality is merely used for testing on the Python side (to try and validate filesystem pickling).
We should decide whether we want to continue supporting filesystem equality and, if so, what the semantics are (is approximate equality useful?).
**Reporter**: [Antoine Pitrou](https://issues.apache.org/jira/browse/ARROW-16928) / @pitrou
**Note**: *This issue was originally created as [ARROW-16928](https://issues.apache.org/jira/browse/ARROW-16928). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reviewing the filesystem equality implementations, especially S3Options::Equals, and the Python-side tests that use equality to validate filesystem pickling. Compare the cost and reliability of the current behavior across filesystem types. Done means the project has settled and documented equality semantics, with corresponding tests updated to reflect that decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend-api-design
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100