apache / apache/datafusion

DataFusion should support casting strings such as "4e7" to decimal

Open
#10,315 4 comments 5 reactions 0 assignees View on GitHub
enhancement waiting-on-upstream
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

DataFusion supports casting the string `4e7` to float but not to decimal. This is inconsistent with Postgres (and Apache Spark).

## Postgres
```
postgres=# select cast('4e7' as float);
float8
----------
40000000
(1 row)

postgres=# select cast('4e7' as decimal(10,2));
numeric
-------------
40000000.00
```

## Apache Spark

```
scala> spark.sql("select cast('4e7' as float)").show
+------------------+
|CAST(4e7 AS FLOAT)|
+------------------+
| 4.0E7|
+------------------+

scala> spark.sql("select cast('4e7' as decimal(10,2))").show
+--------------------------+
|CAST(4e7 AS DECIMAL(10,2))|
+--------------------------+
| 40000000.00|
+--------------------------+
```

## DataFusion

```
DataFusion CLI v37.0.0
❯ select cast('4e7' as float);
+-------------+
| Utf8("4e7") |
+-------------+
| 40000000.0 |
+-------------+
1 row in set. Query took 0.010 seconds.

❯ select cast('4e7' as decimal(10,2));
Arrow error: Cast error: Cannot cast string '4e7' to value of Decimal128(38, 10) type
```

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the `CAST('4e7' AS DECIMAL(10,2))` query in the DataFusion CLI and compare it with the existing float cast behavior. The work is done when scientific-notation strings such as `4e7` cast to decimal successfully and produce the expected value shown for Postgres and Apache Spark.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.