DistCP errors spit to console instead of log
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- aws, hadoop, scala
- Domain
- cli, distributed-systems
Research direction
No repository files or tests are named. Start by tracing the GeoMesa DistCP integration and its Hadoop DistCp error handling, then identify where failures are sent to the console while progress is rendered. Done means DistCP failures are logged without disrupting the progress bar, with coverage for the reported missing S3A file case.
Written by the indexing model from the issue text.
Description
DistCP errors spit to console instead of log
Gross DistCP failures mess up the nice progress bar:
INFO Submitting job - please wait... [33/310] INFO Tracking available at http://ip-10-229-67-60.ec2.internal:20888/proxy/application_1500571557669_0001/ Map (stage 1/3): [============================================================] 100% complete 231735516 mapped 0 failed in 00:24:30 Reduce (stage 2/3): [============================================================] 100% complete 231735516 ingested in 00:08:21 INFO Submitting distcp job - please wait... Error: java.io.FileNotFoundException: No such file or directory: s3a://------------/parquet/testrel20bit/activity/2016/12/27/0583312_0000.parquet at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1004) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77) at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:203) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:257) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143
Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-1956
Key: GEOMESA-1956
Type: Improvement
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Andrew Hulbert
Created: Thu, 20 Jul 2017 14:43:57 -0400
Updated: Thu, 20 Jul 2017 14:43:57 -0400
Parent: GEOMESA-1905
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 446
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 31
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from locationtech/geomesa
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
locationtech/geomesa#10761 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
locationtech/geomesa#10324 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
locationtech/geomesa#10111 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
locationtech/geomesa#10052 ·
-
Difficulty 1/5 1-3 hours Newbie friendliness 78/100
locationtech/geomesa#9998 ·
All issues in locationtech/geomesa
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
requires-triage
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/datafusion-comet#6060 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
lichess-org/lila#21756 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
salesforce/evalon#10 ·