stackabletech / stackabletech/docker-images
Removing MapReduce from Hadoop causes demo job to fail
Nobody has claimed this yet.
- Dominant language
- Dockerfile
- Stars
- 22
- Forks
- 6
- Avg merge
- 16h 59m
- Merged PRs (30d)
- 30
Description
In the hbase-hdfs-load-cycling-data we use a distcp command to copy data in the demo.
This worked prior to 24.7 release, since then we removed the map reduce component which makes this job fail.
Error: Unable to initialize main class org.apache.hadoop.tools.DistCp
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/Job
Solution in the demo is fallback on a 24.3 image, but we need a general solution in terms of shipping with/without map reduce or provide e.g. custom images for demos.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting the hbase-hdfs-load-cycling-data demo and the image configuration used for the distcp command. Reproduce the failure with the 24.7 image and compare it with 24.3, then determine the supported packaging or custom-image approach for demos with and without MapReduce. Done means the demo job runs without the missing Job class error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- dockerfile, hadoop
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100