alibaba / alibaba/DataX

datax向hbase同步数据报错:com.alibaba.datax.common.exception.DataXException: Code:[Hbasewriter-12], Description:[获取hbase BufferedMutator 时出错.]. - org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations

Open
#1,259 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

数据量6000的导入没有问题,导入数据7万条时,程序报错,错误内容如下:
```ERROR Engine -

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Hbasewriter-12], Description:[获取hbase BufferedMutator 时出错.]. - org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:312)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:155)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:392)
at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xHelper.checkHbaseTable(Hbase11xHelper.java:186)
at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xHelper.getBufferedMutator(Hbase11xHelper.java:92)
at com.alibaba.datax.plugin.writer.hbase11xwriter.HbaseAbstractTask.(HbaseAbstractTask.java:38)
at com.alibaba.datax.plugin.writer.hbase11xwriter.NormalTask.(NormalTask.java:27)
at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xWriter$Task.init(Hbase11xWriter.java:58)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44)
at java.lang.Thread.run(Thread.java:745)
- org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:312)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:155)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:392)
at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xHelper.checkHbaseTable(Hbase11xHelper.java:186)
at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xHelper.getBufferedMutator(Hbase11xHelper.java:92)
at com.alibaba.datax.plugin.writer.hbase11xwriter.HbaseAbstractTask.(HbaseAbstractTask.java:38)
at com.alibaba.datax.plugin.writer.hbase11xwriter.NormalTask.(NormalTask.java:27)
at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xWriter$Task.init(Hbase11xWriter.java:58)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44)
at java.lang.Thread.run(Thread.java:745)
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:40)
at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xHelper.getBufferedMutator(Hbase11xHelper.java:102)
at com.alibaba.datax.plugin.writer.hbase11xwriter.HbaseAbstractTask.(HbaseAbstractTask.java:38)
at com.alibaba.datax.plugin.writer.hbase11xwriter.NormalTask.(NormalTask.java:27)
at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xWriter$Task.init(Hbase11xWriter.java:58)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:312)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:155)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:392)
at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xHelper.checkHbaseTable(Hbase11xHelper.java:186)
at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xHelper.getBufferedMutator(Hbase11xHelper.java:92)
... 5 more```

Contributor guide

No contributing guide indexed for this repository

Research direction

The failure path is named in the stack trace: Hbase11xHelper.checkHbaseTable and getBufferedMutator, followed by HBase client region-location calls. Start by reproducing the 70,000-row import and inspect the HBase writer and client configuration around that path. Done means identifying the cause of the region-location failure and verifying a resolution or documenting the required configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.