alibaba / alibaba/DataX

elasticsearch报的一堆网络错误

Open
#237 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

1.org.apache.http.NoHttpResponseException: 192.168.100.98:9200 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) ~[httpcore-4.4.jar:4.4]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) ~[httpcore-4.4.jar:4.4]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167) ~[httpclient-4.4.jar:4.4]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) ~[httpcore-4.4.jar:4.4]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) ~[httpcore-4.4.jar:4.4]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.4.jar:4.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.4.jar:4.4]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:115) ~[jest-2.4.0.jar:na]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:57) ~[jest-2.4.0.jar:na]
at com.alibaba.datax.plugin.writer.elasticsearchwriter.ESClient.bulkInsert(ESClient.java:220) ~[elasticsearchwriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.elasticsearchwriter.ESWriter$Task$2.call(ESWriter.java:400) ~[elasticsearchwriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.elasticsearchwriter.ESWriter$Task$2.call(ESWriter.java:397) ~[elasticsearchwriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.common.util.RetryUtil$Retry.call(RetryUtil.java:164) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.common.util.RetryUtil$Retry.doRetry(RetryUtil.java:111) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.common.util.RetryUtil.executeWithRetry(RetryUtil.java:30) [datax-common-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.elasticsearchwriter.ESWriter$Task.doBatchInsert(ESWriter.java:397) [elasticsearchwriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.elasticsearchwriter.ESWriter$Task.startWrite(ESWriter.java:277) [elasticsearchwriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) [datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_181]
这要怎么解决。我是取1000个数据 往一个远程的elasticsearch写数据,network.host已经改成0.0.0.0了,这个还是我设置"batchSize": 10的结果,如果默认的1000,就直接报fail to response这个异常。总之就是网络错误,大佬们,怎么解决?貌似是jest的问题?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with com.alibaba.datax.plugin.writer.elasticsearchwriter.ESClient.bulkInsert and ESWriter$Task.doBatchInsert, then trace the RetryUtil calls shown in the stack trace. Reproduce the remote Elasticsearch write with batchSize 10 and 1000 while checking the Jest HTTP client path. Done means the failure cause is identified and the writer handles or reports it reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.