linkedin / linkedin/dr-elephant
Unable to deploy dr elephant for hadoop 2.6.0
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.4k
- Forks
- 839
- PR merge metrics
- No merged PRs in 30d
Description
we dont have spark in our environment ,so my compile.conf looks like --
Hadoop Version : 2.6.0
Spark Version : 1.4.0
Other opts set : -Dsbt.repository.config=app-conf/resolver.conf
ERROR --
[info] 12 tests, 0 failures, 0 errors
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] com.linkedin.drelephant.util.InfoExtractorTest
[info] + testGetSchedulerInstanceAirflow
[info] + testLoadInfoMapReduce
[info] + testGetSchedulerInstanceAzkaban
[info] + testLoadInfoSpark
[info] + testGetSchedulerInstanceNull
[info] + testLoadInfoSparkNoConfig
[info] + testLoadSchedulerInfo
[info] + testGetSchedulerInstanceOozie
[info]
[info]
[info] Total for test com.linkedin.drelephant.util.InfoExtractorTest
[info] Finished in 0.001 seconds
[info] 8 tests, 0 failures, 0 errors
[info] com.linkedin.drelephant.schedulers.OozieSchedulerTest
[info] + testDepthCalculation
[info] + testOozieLoadInfoWithOozieClientException
[info] + testOozieLoadsNameFromConfData
[info] + testManualCommittedJobAppNameUnique
[info] + testOozieLoadInfoWithNullProperty
[info] + testOozieScheduledJob
[info] + testManualCommittedJob
[info] + testOozieLoadInfoWithMissingProperty
[info] + tesOozieScheduledJobAppNameUnique
[info] + testUserGivenTemplateArePreferredUrl
[info]
[info]
[info] Total for test com.linkedin.drelephant.schedulers.OozieSchedulerTest
[info] Finished in 0.002 seconds
[info] 10 tests, 0 failures, 0 errors
[info] ScalaTest
[info] Run completed in 46 seconds, 235 milliseconds.
[info] Total number of tests run: 147
[info] Suites: completed 15, aborted 0
[info] Tests: succeeded 141, failed 6, canceled 0, ignored 0, pending 0
[info] *** 6 TESTS FAILED ***
[error] Failed: Total 377, Failed 6, Errors 0, Passed 370, Skipped 1
[error] Failed tests:
[error] com.linkedin.drelephant.util.SparkUtilsTest
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 241 s, completed Jan 29, 2018 6:05:00 PM
- cd target/universal
./compile.sh: line 131: cd: target/universal: No such file or directory
++ /bin/ls '*.zip'
/bin/ls: cannot access *.zip: No such file or directory - ZIP_NAME=
- unzip
UnZip 6.00 of 20 April 2009, by Info-ZIP. Maintained by C. Spieler. Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.
Usage: unzip [-Z] [-opts[modifiers]] file[.zip] [list] [-x xlist] [-d exdir]
Default action is to extract files in list, except those in xlist, to exdir;
file[.zip] may be a wildcard. -Z => ZipInfo mode ("unzip -Z" for usage).
-p extract files to pipe, no messages -l list files (short format)
-f freshen existing files, create none -t test compressed archive data
-u update files, create if necessary -z display archive comment only
-v list verbosely/show version info -T timestamp archive to latest
-x exclude files that follow (in xlist) -d extract files into exdir
modifiers:
-n never overwrite existing files -q quiet mode (-qq => quieter)
-o overwrite files WITHOUT prompting -a auto-convert any text files
-j junk paths (do not make directories) -aa treat ALL files as text
-U use escapes for all non-ASCII Unicode -UU ignore any Unicode fields
-C match filenames case-insensitively -L make (some) names lowercase
-X restore UID/GID info -V retain VMS version numbers
-K keep setuid/setgid/tacky permissions -M pipe through "more" pager
-O CHARSET specify a character encoding for DOS, Windows and OS/2 archives
-I CHARSET specify a character encoding for UNIX and other archives
See "unzip -hh" or unzip.txt for more help. Examples:
unzip data1 -x joe => extract all files except joe from zipfile data1.zip
unzip -p foo | more => send contents of foo.zip via pipe into program more
unzip -fo foo ReadMe => quietly replace existing ReadMe if archive file newer
- rm
rm: missing operand
Try `rm --help' for more information. - DIST_NAME=
- chmod +x /bin/dr-elephant
chmod: cannot access `/bin/dr-elephant': No such file or directory - sed -i.bak '/declare -r app_classpath/s/.$/:
hadoop classpath:${ELEPHANT_CONF_DIR}"/' /bin/dr-elephant
sed: can't read /bin/dr-elephant: No such file or directory - cp /home/mnarang/dr-elephant/scripts/start.sh /bin/
cp: cannot create regular file `/bin/start.sh': Permission denied - cp /home/mnarang/dr-elephant/scripts/stop.sh /bin/
cp: cannot create regular file `/bin/stop.sh': Permission denied - cp -r /home/mnarang/dr-elephant/app-conf
cp: missing destination file operand after/home/mnarang/dr-elephant/app-conf' Trycp --help' for more information. - zip -r .zip
zip error: Nothing to do! (.zip)
- mv .zip /home/mnarang/dr-elephant/dist/
mv: cannot stat `.zip': No such file or directory
Can you guys please help ?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the build from compile.conf and compile.sh, then inspect the six failures reported in com.linkedin.drelephant.util.SparkUtilsTest. Check why target/universal is not produced and whether the Hadoop 2.6.0 and Spark 1.4.0 settings are supported; done means the tests pass and the distribution archive is generated successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, scala, spark
- Domain
- build-system, data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100