linkedin / linkedin/dr-elephant

Unable to deploy dr elephant for hadoop 2.6.0

Open
#325 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.4k
Forks
839
PR merge metrics
No merged PRs in 30d

Description

we dont have spark in our environment ,so my compile.conf looks like --

Hadoop Version : 2.6.0
Spark Version : 1.4.0
Other opts set : -Dsbt.repository.config=app-conf/resolver.conf

ERROR --

[info] 12 tests, 0 failures, 0 errors
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] application - Starting Application...
[info] application - Stopping application...
[info] com.linkedin.drelephant.util.InfoExtractorTest
[info] + testGetSchedulerInstanceAirflow
[info] + testLoadInfoMapReduce
[info] + testGetSchedulerInstanceAzkaban
[info] + testLoadInfoSpark
[info] + testGetSchedulerInstanceNull
[info] + testLoadInfoSparkNoConfig
[info] + testLoadSchedulerInfo
[info] + testGetSchedulerInstanceOozie
[info]
[info]
[info] Total for test com.linkedin.drelephant.util.InfoExtractorTest
[info] Finished in 0.001 seconds
[info] 8 tests, 0 failures, 0 errors
[info] com.linkedin.drelephant.schedulers.OozieSchedulerTest
[info] + testDepthCalculation
[info] + testOozieLoadInfoWithOozieClientException
[info] + testOozieLoadsNameFromConfData
[info] + testManualCommittedJobAppNameUnique
[info] + testOozieLoadInfoWithNullProperty
[info] + testOozieScheduledJob
[info] + testManualCommittedJob
[info] + testOozieLoadInfoWithMissingProperty
[info] + tesOozieScheduledJobAppNameUnique
[info] + testUserGivenTemplateArePreferredUrl
[info]
[info]
[info] Total for test com.linkedin.drelephant.schedulers.OozieSchedulerTest
[info] Finished in 0.002 seconds
[info] 10 tests, 0 failures, 0 errors
[info] ScalaTest
[info] Run completed in 46 seconds, 235 milliseconds.
[info] Total number of tests run: 147
[info] Suites: completed 15, aborted 0
[info] Tests: succeeded 141, failed 6, canceled 0, ignored 0, pending 0
[info] *** 6 TESTS FAILED ***
[error] Failed: Total 377, Failed 6, Errors 0, Passed 370, Skipped 1
[error] Failed tests:
[error] com.linkedin.drelephant.util.SparkUtilsTest
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 241 s, completed Jan 29, 2018 6:05:00 PM

  • cd target/universal
    ./compile.sh: line 131: cd: target/universal: No such file or directory
    ++ /bin/ls '*.zip'
    /bin/ls: cannot access *.zip: No such file or directory
  • ZIP_NAME=
  • unzip
    UnZip 6.00 of 20 April 2009, by Info-ZIP. Maintained by C. Spieler. Send
    bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Usage: unzip [-Z] [-opts[modifiers]] file[.zip] [list] [-x xlist] [-d exdir]
Default action is to extract files in list, except those in xlist, to exdir;
file[.zip] may be a wildcard. -Z => ZipInfo mode ("unzip -Z" for usage).

-p extract files to pipe, no messages -l list files (short format)
-f freshen existing files, create none -t test compressed archive data
-u update files, create if necessary -z display archive comment only
-v list verbosely/show version info -T timestamp archive to latest
-x exclude files that follow (in xlist) -d extract files into exdir
modifiers:
-n never overwrite existing files -q quiet mode (-qq => quieter)
-o overwrite files WITHOUT prompting -a auto-convert any text files
-j junk paths (do not make directories) -aa treat ALL files as text
-U use escapes for all non-ASCII Unicode -UU ignore any Unicode fields
-C match filenames case-insensitively -L make (some) names lowercase
-X restore UID/GID info -V retain VMS version numbers
-K keep setuid/setgid/tacky permissions -M pipe through "more" pager
-O CHARSET specify a character encoding for DOS, Windows and OS/2 archives
-I CHARSET specify a character encoding for UNIX and other archives

See "unzip -hh" or unzip.txt for more help. Examples:
unzip data1 -x joe => extract all files except joe from zipfile data1.zip
unzip -p foo | more => send contents of foo.zip via pipe into program more
unzip -fo foo ReadMe => quietly replace existing ReadMe if archive file newer

  • rm
    rm: missing operand
    Try `rm --help' for more information.
  • DIST_NAME=
  • chmod +x /bin/dr-elephant
    chmod: cannot access `/bin/dr-elephant': No such file or directory
  • sed -i.bak '/declare -r app_classpath/s/.$/:hadoop classpath:${ELEPHANT_CONF_DIR}"/' /bin/dr-elephant
    sed: can't read /bin/dr-elephant: No such file or directory
  • cp /home/mnarang/dr-elephant/scripts/start.sh /bin/
    cp: cannot create regular file `/bin/start.sh': Permission denied
  • cp /home/mnarang/dr-elephant/scripts/stop.sh /bin/
    cp: cannot create regular file `/bin/stop.sh': Permission denied
  • cp -r /home/mnarang/dr-elephant/app-conf
    cp: missing destination file operand after /home/mnarang/dr-elephant/app-conf' Try cp --help' for more information.
  • zip -r .zip

zip error: Nothing to do! (.zip)

  • mv .zip /home/mnarang/dr-elephant/dist/
    mv: cannot stat `.zip': No such file or directory

Can you guys please help ?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the build from compile.conf and compile.sh, then inspect the six failures reported in com.linkedin.drelephant.util.SparkUtilsTest. Check why target/universal is not produced and whether the Hadoop 2.6.0 and Spark 1.4.0 settings are supported; done means the tests pass and the distribution archive is generated successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, scala, spark
Domain
build-system, data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.