linkedin / linkedin/dr-elephant

setup dr-elephant on centos7 with hadoop 2.7.3 and Spark Version 1.6.3

Open
#320 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.4k
Forks
839
PR merge metrics
No merged PRs in 30d

Description

Hi,

do you have some ideas please ?

could we setup Dr-elephant on centos7 with hadoop 2.7.3 and Spark Version 1.6.3 ?
i am trying to do it but i had some error to compile it ?
i am fallowing https://github.com/linkedin/dr-elephant/wiki/Quick-Setup-Instructions.
and the error with the compile dr-elephant is :

[info] - returns the job failure rate *** FAILED ***
[info] "0[,]400" was not equal to "0[.]400" (JobsHeuristicTest.scala:74)
[info] - returns the list of jobs with high task failure rates *** FAILED ***
[info] "...task failure rate: 0[,600)
[info] job 4, zzz (task failure rate: 0,]800)" was not equal to "...task failure rate: 0[.600)
[info] job 4, zzz (task failure rate: 0.]800)" (JobsHeuristicTest.scala:78)

[info] *** 13 TESTS FAILED ***
[error] Failed: Total 377, Failed 17, Errors 0, Passed 359, Skipped 1
[error] Failed tests:
[error] com.linkedin.drelephant.spark.heuristics.ExecutorsHeuristicTest
[error] com.linkedin.drelephant.spark.heuristics.StagesHeuristicTest
[error] com.linkedin.drelephant.util.MemoryFormatUtilsTest
[error] com.linkedin.drelephant.spark.heuristics.JobsHeuristicTest
[error] com.linkedin.drelephant.util.SparkUtilsTest
[error] com.linkedin.drelephant.util.UtilsTest
[error] com.linkedin.drelephant.math.StatisticsTest
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 572 s, completed 10 janv. 2018 23:00:46
./compile.sh: ligne 131 : cd: target/universal: Aucun fichier ou dossier de ce type
/bin/ls: impossible d'accéder à *.zip: Aucun fichier ou dossier de ce type
UnZip 6.00 of 20 April 2009, by Info-ZIP. Maintained by C. Spieler. Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Usage: unzip [-Z] [-opts[modifiers]] file[.zip] [list] [-x xlist] [-d exdir]
Default action is to extract files in list, except those in xlist, to exdir;
file[.zip] may be a wildcard. -Z => ZipInfo mode ("unzip -Z" for usage).

-p extract files to pipe, no messages -l list files (short format)
-f freshen existing files, create none -t test compressed archive data
-u update files, create if necessary -z display archive comment only
-v list verbosely/show version info -T timestamp archive to latest
-x exclude files that follow (in xlist) -d extract files into exdir
modifiers:
-n never overwrite existing files -q quiet mode (-qq => quieter)
-o overwrite files WITHOUT prompting -a auto-convert any text files
-j junk paths (do not make directories) -aa treat ALL files as text
-U use escapes for all non-ASCII Unicode -UU ignore any Unicode fields
-C match filenames case-insensitively -L make (some) names lowercase
-X restore UID/GID info -V retain VMS version numbers
-K keep setuid/setgid/tacky permissions -M pipe through "more" pager
-O CHARSET specify a character encoding for DOS, Windows and OS/2 archives
-I CHARSET specify a character encoding for UNIX and other archives

See "unzip -hh" or unzip.txt for more help. Examples:
unzip data1 -x joe => extract all files except joe from zipfile data1.zip
unzip -p foo | more => send contents of foo.zip via pipe into program more
unzip -fo foo ReadMe => quietly replace existing ReadMe if archive file newer
rm: opérande manquant
Saisissez « rm --help » pour plus d'informations.
chmod: impossible d'accéder à « /bin/dr-elephant »: Aucun fichier ou dossier de ce type
sed: impossible de lire /bin/dr-elephant: Aucun fichier ou dossier de ce type
cp: opérande de fichier cible manquant après « /root/dr-elephant/app-conf »
Saisissez « cp --help » pour plus d'informations.

zip error: Nothing to do! (.zip)
mv: impossible d'évaluer « .zip »: Aucun fichier ou dossier de ce type

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with compile.sh and the listed failed test classes, especially JobsHeuristicTest.scala, to reproduce the CentOS 7 build failure with Hadoop 2.7.3 and Spark 1.6.3. Check why the tests produce comma decimal separators and why target/universal is not created. Done means the test suite passes and compile.sh produces the expected archive.

Written by the indexing model from the issue text.

Assessment

Tech stack
centos, hadoop, scala
Domain
build-system, distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.