apache / apache/hudi

org.apache.hudi.sink.TestWriteCopyOnWrite will failed when local hadoop env exists

Open
#15,298 1 comment 0 reactions 0 assignees View on GitHub
area:tests from-jira priority:high status:pr-available
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

org.apache.hudi.exception.HoodieException: Error while checking whether table exists under path:/var/folders/yy/xxx/T/junit8015457800939027663

    at org.apache.hudi.util.StreamerUtil.tableExists(StreamerUtil.java:324)
    at org.apache.hudi.util.StreamerUtil.initTableIfNotExists(StreamerUtil.java:289)
    at org.apache.hudi.util.StreamerUtil.initTableIfNotExists(StreamerUtil.java:276)
    at org.apache.hudi.sink.StreamWriteOperatorCoordinator.start(StreamWriteOperatorCoordinator.java:184)
    at org.apache.hudi.sink.utils.StreamWriteFunctionWrapper.openFunction(StreamWriteFunctionWrapper.java:133)
    at org.apache.hudi.sink.utils.TestWriteBase$TestHarness.preparePipeline(TestWriteBase.java:158)
    at org.apache.hudi.sink.utils.TestWriteBase$TestHarness.preparePipeline(TestWriteBase.java:146)
    at org.apache.hudi.sink.TestWriteCopyOnWrite.preparePipeline(TestWriteCopyOnWrite.java:449)
    at
Caused by: java.net.ConnectException: Call From xxx/127.0.0.1 to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:827)

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4461
- Type: Test

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with org.apache.hudi.sink.TestWriteCopyOnWrite.preparePipeline and the setup in TestWriteBase$TestHarness.preparePipeline. Trace the StreamWriteOperatorCoordinator.start call into StreamerUtil.tableExists and reproduce the failure with a local Hadoop environment present. Done means the test no longer fails with a connection refusal to localhost:8020 in that environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.