apache / apache/beam

Error: "Unable to find registrar for hdfs" - need to prevent/improve error message

Open
#18,466 0 comments 0 reactions 0 assignees View on GitHub
core improvement java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

I've noticed a number of user reports where jobs are failing with the error message "Unable to find registrar for hdfs":
* https://stackoverflow.com/questions/44497662/apache-beamunable-to-find-registrar-for-hdfs/44508533?noredirect=1#comment76026835_44508533
* https://lists.apache.org/thread.html/144c384e54a141646fcbe854226bb3668da091c5dc7fa2d471626e9b@%3Cuser.beam.apache.org%3E
* https://lists.apache.org/thread.html/e4d5ac744367f9d036a1f776bba31b9c4fe377d8f11a4b530be9f829@%3Cuser.beam.apache.org%3E

This isn't too many reports, but it is the only time I can recall so many users reporting the same error message in a such a short amount of time.

We believe the problem is one of two things:
1) bad uber jar creation
2) incorrect HDFS configuration

However, it's highly possible this could have some other root cause.

It seems like it'd be useful to:
1) Follow up with the above reports to see if they've resolved the issue, and if so what fixed it. There may be another root cause out there.
2) Improve the error message to include more information about how to resolve it
3) See if we can improve detection of the error cases to give more specific information (specifically, if HDFS is miconfigured, can we detect that somehow and tell the user exactly that?)
4) update documentation

Imported from Jira [BEAM-2457](https://issues.apache.org/jira/browse/BEAM-2457). Original Jira may contain additional context.
Reported by: sisk.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked user reports and the original Jira issue to determine whether bad uber jars, HDFS configuration, or another cause produces this error. Then locate the error handling and related documentation, and define completion as clearer diagnosis or messaging, improved detection where feasible, and updated guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.