apache / apache/beam

Pipeline fails to run remotely

Open
#19,874 0 comments 0 reactions 0 assignees View on GitHub
bug core dataflow P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Beam fails to run remote pipeline when the class loader has URLs other than _[file://_](file://_/)._ The environment uses the [Manifold project](http://manifold.systems), which installs a custom URL handler of the form: _manifoldclass://_, which contributes to the URL classpath, however Beam throws an exception when it encounters this URL. Instead Beam should *ignore* or at most warn about URLs it does not recognize.

A similar issue was recently fixed in the Spring framework: [https://github.com/spring-projects/spring-framework/issues/21950](https://github.com/spring-projects/spring-framework/issues/21950). Beam should follow suit.

See external Manifold issue: [https://github.com/manifold-systems/manifold/issues/143](https://github.com/manifold-systems/manifold/issues/143)

 

Imported from Jira [BEAM-8875](https://issues.apache.org/jira/browse/BEAM-8875). Original Jira may contain additional context.
Reported by: rsmckinney.

Contributor guide

Open the contributing guide

Research direction

Start by locating Beam's remote pipeline classpath handling and the code that rejects non-file URL schemes; compare the expected behavior with the linked Spring Framework issue. Done means a class loader containing a custom URL such as manifoldclass:// no longer causes the remote pipeline to fail, and it is ignored or warned about.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.