apache / apache/iotdb

批量写入报 TSStatus(code:500)

Ouverte
#4,639 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Java
Étoiles
6.4k
Forks
1.2k
Merge moyen
1 j 23 h
PR mergées (30 j)
115

Description

**Describe the bug**
从 mysql 迁移数据,开始每次从数据迁移1000条记录,开始正常,后来我就改为 1万条,就开始报这个错,我再改为200条都报
TSStatus(code:500)

**To Reproduce**
Steps to reproduce the behavior:
1. 从mysql 查询1000条记录,每条记录60个字段,for 循环插入 iotdb
2. 开始测试了3个表正常,数据都迁移完成
3. 上线到正式平台,每次迁移1000条,发现要迁移4500个表要很久
4. 加大到每次迁移1万条,iotdb 开始报错 TSStatus(code:500)
5. 改回到1000条,继续报错 TSStatus(code:500)
6. 改回到200条,继续报错 TSStatus(code:500)
7. 插入代码:
public void saveTracks(List tracks) {
if(tracks !=null && tracks.size() > 0)
{
log.error("saveTracks size:" + tracks.size());
Map tablets = new HashMap();
for(int i = 0; i < tracks.size(); ++i)
{
Track track = tracks.get(i);
String deviceId = track.getDeviceid();
String tableName = getTableName(deviceId);
Tablet tablet = tablets.get(tableName);
if(tablet == null)
{
tablet = new Tablet(tableName, schemas);
tablet.rowSize = 0;
tablets.put(tableName, tablet);
}

Map entityToMap = EntityUtil.entityToMap(track);
int rowIndex = tablet.rowSize;
tablet.rowSize = rowIndex + 1;
long timestamp = track.getUpdatetime();
tablet.addTimestamp(rowIndex, timestamp);
Iterator> it = entityToMap.entrySet().iterator();
while (it.hasNext()) {
Entry entry = it.next();
String measurementId = entry.getKey();
Object value = entry.getValue();
try
{
Integer indexOfValueObject = measurementIndex.get(measurementId);
if(indexOfValueObject != null && indexOfValueObject >= 0)
{
if(value instanceof String)
{
String valueString = (String) value;
Binary valueBinary = Binary.valueOf(valueString);
tablet.addValue(measurementId, rowIndex, valueBinary);
}
else
{
tablet.addValue(measurementId, rowIndex, value);
}
}
}catch(Exception e)
{
log.error("measurementId:" + measurementId, e);
}
}

}

try {
pool.insertTablets(tablets);
} catch (Exception e) {
e.printStackTrace();
}
}
}
[log_error.zip](https://github.com/apache/iotdb/files/7777944/log_error.zip)

8. 客户端报错:
TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500), TSStatus(code:500)];Batch process failed:[TSStatus(code:500)];Batch process failed:[TSStatus(code:500), TSStatus(code:500)];
at org.apache.iotdb.session.Session.insertByGroup(Session.java:1734)
at org.apache.iotdb.session.Session.insertTabletsWithLeaderCache(Session.java:1302)
at org.apache.iotdb.session.Session.insertTablets(Session.java:1281)
at org.apache.iotdb.session.pool.SessionPool.insertTablets(SessionPool.java:478)
at org.apache.iotdb.session.pool.SessionPool.insertTablets(SessionPool.java:465)
at com.minigps.iotdb.IotDBMgr.saveTracks(IotDBMgr.java:1117)
9. 服务器日志:
2021-12-27 02:06:13,072 [pool-14-IoTDB-RPC-Client-106] ERROR o.a.i.d.e.s.StorageGroupProcessor:961 - insert to TsFileProcessor error
org.apache.iotdb.db.exception.WriteProcessException: java.lang.NullPointerException
at org.apache.iotdb.db.engine.storagegroup.TsFileProcessor.insertTablet(TsFileProcessor.java:273)
at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.insertTabletToTsFileProcessor(StorageGroupProcessor.java:956)
at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.insertTablet(StorageGroupProcessor.java:898)
at org.apache.iotdb.db.engine.StorageEngine.insertTablet(StorageEngine.java:706)
at org.apache.iotdb.db.qp.executor.PlanExecutor.insertTablet(PlanExecutor.java:1432)
at org.apache.iotdb.db.qp.executor.PlanExecutor.insertTablet(PlanExecutor.java:1411)
at org.apache.iotdb.db.qp.executor.PlanExecutor.processNonQuery(PlanExecutor.java:258)
at org.apache.iotdb.db.service.TSServiceImpl.executeNonQuery(TSServiceImpl.java:1153)
at org.apache.iotdb.db.service.TSServiceImpl.executeNonQueryPlan(TSServiceImpl.java:1891)
at org.apache.iotdb.db.service.TSServiceImpl.insertTabletsInternal(TSServiceImpl.java:1608)
at org.apache.iotdb.db.service.TSServiceImpl.insertTablets(TSServiceImpl.java:1564)
at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablets.getResult(TSIService.java:2848)
at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablets.getResult(TSIService.java:2828)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

**Expected behavior**
能够正常迁移1万一次数据

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
- OS: Ubuntu 16.04
- Version: iotdb server 0.12.4 客户端 java:0.12.3 jar

**Additional context**
服务器日志文件,见附件
[log_error.zip](https://github.com/apache/iotdb/files/7777945/log_error.zip)

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Commencez par la stack trace du serveur à TsFileProcessor.java:273 et suivez le chemin de insertTablets à travers StorageGroupProcessor.java et StorageEngine.java. Reproduisez la migration avec le serveur IoTDB 0.12.4 et le client Java 0.12.3 indiqués, en faisant varier les tailles de lot et les logs fournis. C’est terminé lorsque l’insert signalé ne produit plus de NullPointerException ni de TSStatus(code:500).

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
java
Domaine
databases
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
Calme
Clarté
Plutôt claire
Accessibilité débutants
35/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.