apache / apache/pinot

RFE: Pinot Java client: option to avoid converting BigDecimal to double when serializing results into JSON

Open
#10,743 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

We are seeing some different behaviors between 0.9.0 and latest Pinot:

Some of our queries are selecting on bytesToBigDecimal(amount_decimal_hex) where amount_decimal_hex is BYTES column, values are hex strings like `001200e1905c5fd123ecb1`, and

I believe the related conversion logic on Pinot is [here](https://github.com/apache/pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/function/scalar/DataTypeConversionFunctions.java#L89).

We're using Pinot java client to make the query, and reading the results with ResultSet.getString , in the particular example, our query to
0.9.0 Pinot got result: 16.253592621780561073, while query to
latest Pinot got result: 16.253592621780562,
notice the difference at the last few digits.

But re-running the same SQL from control UI, both clusters return 16.253592621780561073 , the same results as 0.9.0 returned from Pinot client.

I did further experiment in local IDE with Pinot repo,
```
BigDecimal amount = deserialize(bytes);
System.out.println(hexString + " decoded to " + amount + ", to double:" + amount.doubleValue());
```
outputs
```
001200e1905c5fd123ecb1 decoded to 16.253592621780561073, to double:16.253592621780562
```

It seems somehow the latest Pinot converts BigDecimal to double when serializing results into JSON?
Judging from [getString on ResultTableResultSet](https://github.com/apache/pinot/blob/master/pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/ResultTableResultSet.java#L62) where it gets the value from Jackson JsonNode.
Is there a way we can address this discrepancy?

Contributor guide

Open the contributing guide

Research direction

Start with the conversion logic in pinot-common/src/main/java/org/apache/pinot/common/function/scalar/DataTypeConversionFunctions.java and result handling in pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/ResultTableResultSet.java. Reproduce the BigDecimal value shown in the issue and trace how ResultSet.getString reads the Jackson JsonNode. Done should provide an option that preserves the expected decimal representation when serializing query results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.