apache / apache/doris-spark-connector

[Bug] Read a random table with specific column type 'ipv6' and the value goes with 'x::x' .It will throw an Exeception

Open
#337 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
107
Forks
126
Avg merge
3d 10h
Merged PRs (30d)
4

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Version

25.0.1

### What's Wrong?

When connector reads random table contains Specific "IPV6" column type with such "x :: x" value in it. It will throw a DORISEXCEPTION

### What You Expected?

​Flaw Description:​​

I have identified the root cause of the issue, outlined as follows:

​Scenario Setup:​​
A table is created via JDBC (e.g., using Navicat).
This table contains a column defined with an IPV6 type.
An IPv6 address value like x::x (where x represents hexadecimal digits) is stored in this column.
​Reading Process:​​
The Doris Spark Connector is used to read data from this table.
The read process flows through:

1. DorisPartitionReader.next()
2. AbstractThriftReader.next()
3. hasNext() check
4. Initialization of a RowBatch using the retrieved information.
5. Execution then enters the convertArrowToRowBatch() method within the RowBatch class.

​Problematic Conversion:​​
Within the type-specific switch logic handling the IPV6 column, the following conversion occurs:

String ipv6Str = new String(ipv6VarcharVector.get(rowIndex)); // Convert Arrow vector data to Java String
String ipv6Address = IPUtils.fromBigInteger(new BigInteger(ipv6Str)); // Attempt conversion via BigInteger

The Core Issue:​​ Using new BigInteger(ipv6Str) to parse a string containing :: (like x::x) is problematic. The BigInteger(String) constructor expects a string representing a single, contiguous decimal number. An IPv6 address in compressed format (::) does not conform to this format and will cause a NumberFormatException.

### How to Reproduce?

1. Store an IPv6 address in compressed format (containing ::) within a database column. Ensure the column is mapped to a String type or directly handled as a string during the read process.
2. Use the Doris Spark Connector to read data from this table.
3. When the connector encounters the compressed IPv6 string (x::x) and attempts to parse it using new BigInteger(...), an InputFormat error (specifically, a NumberFormatException) will occur due to the invalid string format for the BigInteger constructor.

### Anything Else?

Why using break clause when encounter NULL value in brach "IPV6" in ROWBACTH? I thought the continue clause may make more sense.Plz guide me in delima.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with RowBatch.convertArrowToRowBatch and follow the IPV6 branch from DorisPartitionReader.next() through AbstractThriftReader.next(). Reproduce the read with a compressed IPv6 value such as x::x and inspect the NULL-handling branch; done means compressed IPv6 values no longer cause a NumberFormatException during connector reads.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.