Create a mini file system to support test
Open
from-jira
priority:high
type:bug
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
MiniDFS cluster causes instability problems for our test, and increases test running time.
We aim to provide a simple file system that support all HDFS api, and utilize the local disk as storage, and hudi file group reader and file group writer to read and write files.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7389
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by examining the tests that use the MiniDFS cluster and the HDFS API surface they require. The work is complete when those tests use a local-disk-backed filesystem with the needed HDFS API behavior and no longer suffer from MiniDFS instability or its test-time overhead.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100