apache / apache/arrow

[C++] Reconsider filesystem equality

Open
#32,247 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: task
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Filesystems support an equality method to compare filesystem instances. The original idea is that all filesystem parameters should be transparent and easily read back, so it should be possible to support equality (similarly, it was envisioned to allow roundtripping filesystems through URIs, though the filesystem-to-URI direction was never implemented).

However, along the way, filesystems like S3 grew increasingly complex and opaque modes of configuration where equality can only be approximated. It can also be costly to compute (for example, `S3Options::Equals` involves fetching the actual secret key and session token, which can take some time: these mere operations consume 5 seconds in the PyArrow test suite).

Right now, filesystem equality is merely used for testing on the Python side (to try and validate filesystem pickling).

We should decide whether we want to continue supporting filesystem equality and, if so, what the semantics are (is approximate equality useful?).

**Reporter**: [Antoine Pitrou](https://issues.apache.org/jira/browse/ARROW-16928) / @pitrou

**Note**: *This issue was originally created as [ARROW-16928](https://issues.apache.org/jira/browse/ARROW-16928). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the filesystem equality implementations, especially S3Options::Equals, and the Python-side tests that use equality to validate filesystem pickling. Compare the cost and reliability of the current behavior across filesystem types. Done means the project has settled and documented equality semantics, with corresponding tests updated to reflect that decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend-api-design
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.