llnl / llnl/magpie

MAGPIE_HOSTNAME_CMD not working with MAGPIE_NO_LOCAL_DIR

Open
#342 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
198
Forks
51
PR merge metrics
No merged PRs in 30d

Description

Hello, we have a setup in our data center that requires both `MAGPIE_HOSTNAME_CMD` and `MAGPIE_NO_LOCAL_DIR` to be used. However, the `no-local-dir` patches use `` myhostname=`hostname` ``, so they do not respect `MAGPIE_HOSTNAME_CMD` or `MAGPIE_HOSTNAME_CMD_MAP`. As a result, there is a mismatch between the config file directories used by Magpie and the ones that e.g. Hadoop or Spark recognize.

I'm willing to contribute a PR to fix this, though I'm not 100% sure what the best approach is. For Hadoop, I think we could add a `magpie_hostname` to the `hadoop-user-functions.sh` and use that for setting `myhostname` in all the setup scripts. For other software (taking Spark as example), I think we can add `MAGPIE_HOSTNAME_CMD` and `MAGPIE_HOSTNAME_CMD_MAP` to `spark-env.sh`, and add a `magpie_hostname` function to each of the setup scripts.

Does that approach make sense to you, @chu11? And if so, would you like me to draft a PR implementing this?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with the no-local-dir patches and the Hadoop hadoop-user-functions.sh and Spark spark-env.sh/setup scripts named in the issue. Trace how myhostname is set, then verify that MAGPIE_HOSTNAME_CMD and MAGPIE_HOSTNAME_CMD_MAP are honored consistently so Magpie, Hadoop, and Spark select matching configuration directories.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, shell, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.