Decimal encoding performance improvement
- Dominant language
- C#
- Stars
- 39
- Forks
- 30
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 16
Description
### Describe the enhancement requested
The current implementation of `DecimalXArray.Builder` shows significantly worse performance compared to other array builders in arrow-dotnet, primarily due to BigInteger arithmetic overhead.
| Method | Mean | Error | StdDev | Gen0 | Gen1 | Gen2 | Allocated |
|------------------------- |------------:|----------:|----------:|-------:|-------:|-------:|----------:|
| Int32ArrayBuilder_Append | 317.7 ns | 8.32 ns | 24.27 ns | 0.0234 | - | - | 296 B |
| Int64ArrayBuilder | 318.1 ns | 9.22 ns | 26.74 ns | 0.0310 | - | - | 392 B |
| StringArrayBuilder | 1,620.8 ns | 32.50 ns | 84.47 ns | 0.1850 | 0.1717 | 0.0916 | 1928 B |
| Decimal32ArrayBuilder | 3,514.0 ns | 68.52 ns | 135.26 ns | 0.0458 | 0.0420 | 0.0153 | 528 B |
| Decimal64ArrayBuilder | 4,400.3 ns | 87.73 ns | 208.50 ns | 0.0992 | - | - | 1256 B |
| Decimal128ArrayBuilder | 5,949.8 ns | 118.44 ns | 216.58 ns | 0.2289 | 0.1068 | 0.0534 | 2456 B |
| Decimal256ArrayBuilder | 10,367.5 ns | 245.93 ns | 713.47 ns | 0.2747 | 0.1526 | 0.1068 | 3184 B |
The results are based on the following benchmark: https://github.com/aik-jahoda/arrow-dotnet/commit/a2cf2a6d0ca65f029fe80c61f0e70e4218178292
###Root cause
The main performance bottleneck comes from reliance on BigInteger arithmetic, which is much slower than using native integer math.
Proposed Directions
I’d like to open a discussion on which approach would best align with arrow-dotnet’s goals and .NET ecosystem support:
### 1. Pre-allocate `BigInteger` powers of 10
Pros:
- Straightforward to implement
- Provides performance gains for all decimal variants across all .NET versions
Cons:
- Only partial improvement, may still lag compared to native integer arithmetic
### 2. Use native integral types (`Int128` in .net 7+, `Int64` in older)
Pros:
- High performance for `Decimal32`, `Decimal64` for all .net versions and `Decimal128` for .net 7+
Cons:
- Performance inconsistency across .NET versions (on pre-.net 7 `Decimal128` must fall back to `BigInteger`)
- `Decimal256` still requires `BigInteger` on all .net versions
### 3. Create/import custom `Int128`/`Int256` types
Pros:
- Consistent performance across all decimal variants
Cons:
- Significant complexity in implementation and maintenance
Contributor guide
Assessment
This issue has not been assessed yet.