develop mechanism to access HDFS over Lustre/networkfs w/o launching a job
- Dominant language
- Shell
- Stars
- 198
- Forks
- 51
- PR merge metrics
- No merged PRs in 30d
Description
I believe through a series of scripting tricks, it would be doable to script this. Hypothetically, launch X hdfs daemons as processes and launch a namenode process on the same node. Configure them to use appropriate paths in lustre/networkfs as their "local drive". Make sure each have separate ports so they communicate to each other on the same node.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by surveying Magpie's Hadoop and filesystem-related scripts, then determine the required process, port, and Lustre/networkfs configuration; done would mean a documented mechanism that accesses HDFS without launching a job.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, shell
- Domain
- data-engineering, distributed-systems, hpc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100