stackabletech / stackabletech/hdfs-operator
HDFS with one NameNode and no JournalNodes (no HA) for Integration Tests
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 53
- Forks
- 9
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 10
Description
Druid uses HDFS for DeepStorage in its unit tests. A new HDFS instance needs to be spun up every time a test is run, and HDFS is a major contributor to the overall test duration, and also causes tests to fail occasionally because of its long startup times.
As I have learned today (thanks to Lars!), HDFS only uses a second NameNode and the JournalNodes for high availability (HA) but can also run without them. I think it would be great to have this. For an intance that only lives a few minutes and might not even see data written to it, the most bare-bones setup should be used.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the integration-test setup that creates HDFS and read how NameNodes and JournalNodes are configured. The change is complete when tests can run with one NameNode and no JournalNodes, while preserving the required HDFS behavior and improving startup time or reliability.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust
- Domain
- distributed-systems, infrastructure, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100