llnl / llnl/magpie

develop mechanism to access HDFS over Lustre/networkfs w/o launching a job

Open
#46 1 comment 0 reactions 0 assignees View on GitHub
PotentialEnhancement
Dominant language
Shell
Stars
198
Forks
51
PR merge metrics
No merged PRs in 30d

Description

I believe through a series of scripting tricks, it would be doable to script this. Hypothetically, launch X hdfs daemons as processes and launch a namenode process on the same node. Configure them to use appropriate paths in lustre/networkfs as their "local drive". Make sure each have separate ports so they communicate to each other on the same node.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by surveying Magpie's Hadoop and filesystem-related scripts, then determine the required process, port, and Lustre/networkfs configuration; done would mean a documented mechanism that accesses HDFS without launching a job.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, shell
Domain
data-engineering, distributed-systems, hpc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.