fsspec / fsspec/filesystem_spec
Change the default value for replication factor in HadoopFileSystem
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Hey there!
The current implementation of HadoopFileSystem assumes replication 3 by default. The code I'm referring to is here: https://github.com/fsspec/filesystem_spec/blob/8463a6a587f722f279e987b3e8508f2c9bba50d2/fsspec/implementations/arrow.py#L257
It would be better, IMO, to just use this parameter value from the configuration file. So, this would mean setting replication=0 by default.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at fsspec/implementations/arrow.py around line 257 and inspect HadoopFileSystem's replication parameter and how configuration values are resolved. Change the default so the configuration supplies replication when no explicit value is given, then verify that the resulting behavior uses the configured value rather than assuming replication 3.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100