apache / apache/iotdb

read empty dataframe when one column result by spark iotdb connector

Open
#2,861 3 comments 0 reactions 0 assignees View on GitHub
Module - Spark
Dominant language
Java
Stars
6.4k
Forks
1.2k
Avg merge
1d 23h
Merged PRs (30d)
115

Description

**环境**
- Iotdb 0.9.3 & 0.11.2
- spark 2.3.3
- spark-iotdb-connector 0.11.2

**问题**
spark读取iotdb数据后,返回一个空的DataFrame对象

**分析**
0.9.3中执行带聚合函数的sql,如`select count(root.ln.wf01.wt01.status) from root`,返回为两列,一列是时间列,一列是值,但是在0.11.2中返回的是一列,但是spark-iotdb-connector中并未处理这种,在生成schame的时候,从2开始遍历,导致只有一列的数据返回空schema,代码如下:
spark-iotdb-connector\src\main\scala\org\apache\iotdb\spark\db\Converter.scala
```
def toSparkSchema(options: IoTDBOptions): StructType = {

Class.forName("org.apache.iotdb.jdbc.IoTDBDriver")
val sqlConn: Connection = DriverManager.getConnection(options.url, options.user, options.password)
val sqlStatement: Statement = sqlConn.createStatement()
val hasResultSet: Boolean = sqlStatement.execute(options.sql)

val fields = new ListBuffer[StructField]()
if (hasResultSet) {
val resultSet: ResultSet = sqlStatement.getResultSet
val resultSetMetaData: ResultSetMetaData = resultSet.getMetaData

val printTimestamp = !resultSet.asInstanceOf[IoTDBJDBCResultSet].isIgnoreTimeStamp
if (printTimestamp) {
fields += StructField(SQLConstant.TIMESTAMP_STR, LongType, nullable = false)
}

val colCount = resultSetMetaData.getColumnCount
for (i <- 2 to colCount) {
.....
}
StructType(fields.toList)
}
else {
StructType(fields)
}
}
```
需要加个startCol变量,来确定是从2开始遍历还是1,如下:
```
var startCol = 1
if (printTimestamp) {
fields += StructField(SQLConstant.TIMESTAMP_STR, LongType, nullable = false)
startCol = 2
}

val colCount = resultSetMetaData.getColumnCount
for (i <- startCol to colCount) {
.....
}
```

Contributor guide

Open the contributing guide

Research direction

Start with spark-iotdb-connector/src/main/scala/org/apache/iotdb/spark/db/Converter.scala and inspect toSparkSchema, then reproduce the reported aggregate query with the listed IoTDB and Spark versions. Verify that a one-column result produces a non-empty Spark schema and DataFrame instead of an empty schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark, sql
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.