apache / apache/iotdb

批量写入报 TSStatus(code:500)

Offen
#4,639 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Java
Sterne
6.4k
Forks
1.2k
Ø Merge
1 T. 23 Std.
Gemergte PRs (30 T.)
115

Beschreibung

**Describe the bug**
从 mysql 迁移数据,开始每次从数据迁移1000条记录,开始正常,后来我就改为 1万条,就开始报这个错,我再改为200条都报
TSStatus(code:500)

**To Reproduce**
Steps to reproduce the behavior:
1. 从mysql 查询1000条记录,每条记录60个字段,for 循环插入 iotdb
2. 开始测试了3个表正常,数据都迁移完成
3. 上线到正式平台,每次迁移1000条,发现要迁移4500个表要很久
4. 加大到每次迁移1万条,iotdb 开始报错 TSStatus(code:500)
5. 改回到1000条,继续报错 TSStatus(code:500)
6. 改回到200条,继续报错 TSStatus(code:500)
7. 插入代码:
public void saveTracks(List tracks) {
if(tracks !=null && tracks.size() > 0)
{
log.error("saveTracks size:" + tracks.size());
Map tablets = new HashMap();
for(int i = 0; i < tracks.size(); ++i)
{
Track track = tracks.get(i);
String deviceId = track.getDeviceid();
String tableName = getTableName(deviceId);
Tablet tablet = tablets.get(tableName);
if(tablet == null)
{
tablet = new Tablet(tableName, schemas);
tablet.rowSize = 0;
tablets.put(tableName, tablet);
}

Map entityToMap = EntityUtil.entityToMap(track);
int rowIndex = tablet.rowSize;
tablet.rowSize = rowIndex + 1;
long timestamp = track.getUpdatetime();
tablet.addTimestamp(rowIndex, timestamp);
Iterator> it = entityToMap.entrySet().iterator();
while (it.hasNext()) {
Entry entry = it.next();
String measurementId = entry.getKey();
Object value = entry.getValue();
try
{
Integer indexOfValueObject = measurementIndex.get(measurementId);
if(indexOfValueObject != null && indexOfValueObject >= 0)
{
if(value instanceof String)
{
String valueString = (String) value;
Binary valueBinary = Binary.valueOf(valueString);
tablet.addValue(measurementId, rowIndex, valueBinary);
}
else
{
tablet.addValue(measurementId, rowIndex, value);
}
}
}catch(Exception e)
{
log.error("measurementId:" + measurementId, e);
}
}

}

try {
pool.insertTablets(tablets);
} catch (Exception e) {
e.printStackTrace();
}
}
}
[log_error.zip](https://github.com/apache/iotdb/files/7777944/log_error.zip)

8. 客户端报错:
TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500)];Batch process failed:[TSStatus(code:500)];Batch process failed:[TSStatus(code:500), TSStatus(code:500)];
at org.apache.iotdb.session.Session.insertByGroup(Session.java:1734)
at org.apache.iotdb.session.Session.insertTabletsWithLeaderCache(Session.java:1302)
at org.apache.iotdb.session.Session.insertTablets(Session.java:1281)
at org.apache.iotdb.session.pool.SessionPool.insertTablets(SessionPool.java:478)
at org.apache.iotdb.session.pool.SessionPool.insertTablets(SessionPool.java:465)
at com.minigps.iotdb.IotDBMgr.saveTracks(IotDBMgr.java:1117)
9. 服务器日志:
2021-12-27 02:06:13,072 [pool-14-IoTDB-RPC-Client-106] ERROR o.a.i.d.e.s.StorageGroupProcessor:961 - insert to TsFileProcessor error
org.apache.iotdb.db.exception.WriteProcessException: java.lang.NullPointerException
at org.apache.iotdb.db.engine.storagegroup.TsFileProcessor.insertTablet(TsFileProcessor.java:273)
at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.insertTabletToTsFileProcessor(StorageGroupProcessor.java:956)
at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.insertTablet(StorageGroupProcessor.java:898)
at org.apache.iotdb.db.engine.StorageEngine.insertTablet(StorageEngine.java:706)
at org.apache.iotdb.db.qp.executor.PlanExecutor.insertTablet(PlanExecutor.java:1432)
at org.apache.iotdb.db.qp.executor.PlanExecutor.insertTablet(PlanExecutor.java:1411)
at org.apache.iotdb.db.qp.executor.PlanExecutor.processNonQuery(PlanExecutor.java:258)
at org.apache.iotdb.db.service.TSServiceImpl.executeNonQuery(TSServiceImpl.java:1153)
at org.apache.iotdb.db.service.TSServiceImpl.executeNonQueryPlan(TSServiceImpl.java:1891)
at org.apache.iotdb.db.service.TSServiceImpl.insertTabletsInternal(TSServiceImpl.java:1608)
at org.apache.iotdb.db.service.TSServiceImpl.insertTablets(TSServiceImpl.java:1564)
at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablets.getResult(TSIService.java:2848)
at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablets.getResult(TSIService.java:2828)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

**Expected behavior**
能够正常迁移1万一次数据

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
- OS: Ubuntu 16.04
- Version: iotdb server 0.12.4 客户端 java:0.12.3 jar

**Additional context**
服务器日志文件,见附件
[log_error.zip](https://github.com/apache/iotdb/files/7777945/log_error.zip)

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginnen Sie mit dem Server-Stacktrace bei TsFileProcessor.java:273 und verfolgen Sie den Pfad von insertTablets durch StorageGroupProcessor.java und StorageEngine.java. Reproduzieren Sie die Migration mit dem gemeldeten IoTDB-0.12.4-Server und dem Java-0.12.3-Client, wobei Sie die Batchgrößen und die bereitgestellten Logs variieren. Als erledigt gilt die Aufgabe, wenn der gemeldete Insert keinen NullPointerException oder TSStatus(code:500) mehr erzeugt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
databases
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.