llnl/magpie

View on GitHub

Magpie contains a number of scripts for running Big Data software in HPC environments, including Hadoop and Spark. There is support for Lustre, Slurm, Moab, Torque. LSF, Flux, and more.

Stars: 198
Forks: 51
Open beginner issues: 0
Indexed issues: 38

Dominant language: Shell
License: GPL-2.0
Last GitHub push: Jan 30, 2025
Latest indexed: Sep 19, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

38 open issues indexed

MAGPIE_HOSTNAME_CMD not working with MAGPIE_NO_LOCAL_DIR Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

llnl/magpie#342 · 3 comments · Opened Nov 24, 2023
rename master branch Open

Difficulty 2/5 1-3 hours Newbie friendliness 25/100

llnl/magpie#323 · Opened Aug 27, 2020
create better tensorflow example / default Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

llnl/magpie#308 · Opened Mar 15, 2019
add node mapping to magpie output Open

Difficulty 3/5 1-2 days Newbie friendliness 38/100

llnl/magpie#306 · 1 comment · Opened Feb 22, 2019
enable spark.worker.cleanup by default Open

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

llnl/magpie#291 · 1 comment · Opened Aug 7, 2018
prefix magpie output with "MAGPIE" or similar Open

Difficulty 3/5 1-2 days Newbie friendliness 30/100

llnl/magpie#290 · Opened Jul 24, 2018
RDMA Hadoop/Spark not working with Slurm submission scripts Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

llnl/magpie#276 · 3 comments · Opened Mar 16, 2018
What i need to do to get a working copy of magpie? Open

Difficulty 4/5 3-5 days Newbie friendliness 20/100

llnl/magpie#275 · 3 comments · Opened Feb 26, 2018
come up with better name than MAGPIE_ONE_TIME_RUN Open

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

llnl/magpie#274 · Opened Feb 13, 2018
spark - option to configure queue size Open

Difficulty 2/5 1-3 hours Newbie friendliness 30/100

llnl/magpie#272 · 1 comment · Opened Feb 6, 2018
configuring local block size for spark if not using hadoop Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

llnl/magpie#270 · Opened Jan 30, 2018
refactor testsuite infrastructure Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

llnl/magpie#256 · Opened Dec 15, 2017
launch magpie w/ command line tool instead of submission script Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

llnl/magpie#255 · 1 comment · Opened Dec 15, 2017
develop command line tool to generate submission files Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

llnl/magpie#254 · Opened Dec 15, 2017
have spark / hadoop log application log files to persistent location Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

llnl/magpie#242 · Opened Aug 1, 2017
make a dumb "networkfs" plugin for hadoop Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

llnl/magpie#241 · Opened Jul 31, 2017
testsuite - supply multiple python version testing Open

Difficulty 4/5 3-5 days Newbie friendliness 20/100

llnl/magpie#240 · Opened Jul 27, 2017
Hadoop Intel Lustre Locking Error on Name Node Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

llnl/magpie#238 · 7 comments · Opened Jul 3, 2017
-Xmx passed twice on command line Open

Difficulty 2/5 1-3 hours Newbie friendliness 45/100

llnl/magpie#237 · Opened Jun 28, 2017
Re-visit estimates on thread counts, process limits, file descriptor limits, etc. Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

llnl/magpie#233 · 2 comments · Opened Mar 25, 2017
support mesos in Magpie Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

llnl/magpie#229 · Opened Nov 8, 2016
support giraph in Magpie Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 15/100

llnl/magpie#228 · Opened Oct 6, 2016
magpie 2.X - Support erasure coding option to terasort Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 25/100

llnl/magpie#221 · 2 comments · Opened Aug 30, 2016
exit code from job script should be preserved Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 38/100

llnl/magpie#218 · 1 comment · Opened Aug 26, 2016
Fix sed portability to non-GNU seds Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 35/100

llnl/magpie#203 · Opened Jul 29, 2016
rework MAGPIE_SHUTDOWN_TIME Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

llnl/magpie#186 · Opened Jul 19, 2016
revisit tachyon (now alluxio) Open

PotentialEnhancement

Difficulty 5/5 Over a week Newbie friendliness 20/100

llnl/magpie#164 · Opened Jul 12, 2016
support mechanism to allow magpie jobs & classic MPI jobs to run at the same time Open

PotentialEnhancement

Difficulty 5/5 Over a week Newbie friendliness 20/100

llnl/magpie#148 · 3 comments · Opened Jun 24, 2016
support multiple HDFS file systems at the same time? Open

PotentialEnhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

llnl/magpie#145 · 1 comment · Opened Jun 22, 2016
Safely shutdown each part Open

PotentialEnhancement

Difficulty 4/5 3-5 days Newbie friendliness 30/100

llnl/magpie#101 · 12 comments · Opened Jan 21, 2016
Standalone Configuration Files Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

llnl/magpie#93 · Opened Dec 9, 2015
Use different exit code numbers for different issues Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

llnl/magpie#80 · 2 comments · Opened Dec 4, 2015
Support some sort of Monitoring Software Open

PotentialEnhancement

Difficulty 5/5 Over a week Newbie friendliness 15/100

llnl/magpie#73 · 2 comments · Opened Oct 1, 2015
option to run daemons on separate nodes Open

PotentialEnhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

llnl/magpie#55 · Opened Aug 25, 2015
develop mechanism to access HDFS over Lustre/networkfs w/o launching a job Open

PotentialEnhancement

Difficulty 5/5 Over a week Newbie friendliness 20/100

llnl/magpie#46 · 1 comment · Opened Aug 12, 2015
detect zookeeper versioning differences between runs and output clearer error message Open

PotentialEnhancement

Difficulty 4/5 3-5 days Newbie friendliness 25/100

llnl/magpie#26 · 2 comments · Opened Jul 2, 2015
Support HDFS Namenode failover Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 15/100

llnl/magpie#12 · Opened Jan 14, 2015
Enhancement proposal: making timeout behavior more dynamic Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

llnl/magpie#9 · 7 comments · Opened Jan 9, 2015

llnl/magpie

Get new issues in your inbox