JuliaParallel / JuliaParallel/DTables.jl
`*join` performance with `NamedTuple` as right table is lower than when it's wrapped in a `DTable`
- Dominant language
- Julia
- Stars
- 81
- Forks
- 2
- Avg merge
- 2h 31m
- Merged PRs (30d)
- 1
Description
noticed this
at first glance there's a ton of allocations, but both are doing the same thing, so that's weird
```julia
julia> d = DTable((a=rand(Int32, 1_000_000).%1000, ), 1_000_0)
DTable with 100 partitions
Tabletype: NamedTuple
julia> d2 = DTable((a=rand(Int32, 10_0).%1000, b=rand(100) ), 10_00)
DTable with 1 partitions
Tabletype: NamedTuple
julia> n2 = fetch(d2)
(a = [758, 846, -266, 860, -342, -795, -165, -714, 517, -881 … 110, -499, -300, 650, 486, -492, 632, -889, -893, 423], b = [0.4474367606114661, 0.41985542687007094, 0.9494639302698825,
0.3019064192644436, 0.2842619673644605, 0.33644378094201766, 0.37412489965948725, 0.37382203879610154, 0.9720419165033913, 0.9233942185630507 … 0.31621236318271906, 0.5842933105627444,
0.39096688865395135, 0.9273050552198202, 0.5482263849758218, 0.9594143140554303, 0.015566368537604847, 0.40198063357487346, 0.9473625812682815, 0.1800731851800842])
julia> @time fetch(innerjoin(d, d2, on=:a))
0.598756 seconds (1.68 M allocations: 112.527 MiB, 72.29% compilation time)
(a = [-293, -178, 134, 177, -853, 511, 985, -720, 486, 181 … -946, -165, 517, 532, 915, 960, -971, 286, 486, 177], b = [0.7311473794219386, 0.3818453188579001, 0.4120712143452663, 0.2838680067429056, 0.76032556294911, 0.9119393018450566, 0.7070032863226784, 0.43326972678995523, 0.5482263849758218, 0.2625750813381811 … 0.16431670006855525, 0.37412489965948725, 0.9720419165033913, 0.08060734012468784, 0.5533171336827942, 0.9539225757859043, 0.19101056560985374, 0.9093868620405196, 0.5482263849758218, 0.2838680067429056])
julia> @time fetch(innerjoin(d, d2, on=:a))
0.148911 seconds (614.39 k allocations: 55.040 MiB, 3.89% compilation time)
(a = [-293, -178, 134, 177, -853, 511, 985, -720, 486, 181 … -946, -165, 517, 532, 915, 960, -971, 286, 486, 177], b = [0.7311473794219386, 0.3818453188579001, 0.4120712143452663, 0.2838680067429056, 0.76032556294911, 0.9119393018450566, 0.7070032863226784, 0.43326972678995523, 0.5482263849758218, 0.2625750813381811 … 0.16431670006855525, 0.37412489965948725, 0.9720419165033913, 0.08060734012468784, 0.5533171336827942, 0.9539225757859043, 0.19101056560985374, 0.9093868620405196, 0.5482263849758218, 0.2838680067429056])
julia> @time fetch(innerjoin(d, n2, on=:a))
4.423514 seconds (200.78 M allocations: 7.510 GiB, 44.16% gc time)
(a = [-293, -178, 134, 177, -853, 511, 985, -720, 486, 181 … -946, -165, 517, 532, 915, 960, -971, 286, 486, 177], b = [0.7311473794219386, 0.3818453188579001, 0.4120712143452663, 0.2838680067429056, 0.76032556294911, 0.9119393018450566, 0.7070032863226784, 0.43326972678995523, 0.5482263849758218, 0.2625750813381811 … 0.16431670006855525, 0.37412489965948725, 0.9720419165033913, 0.08060734012468784, 0.5533171336827942, 0.9539225757859043, 0.19101056560985374, 0.9093868620405196, 0.5482263849758218, 0.2838680067429056])
julia> @time fetch(innerjoin(d, n2, on=:a))
4.389631 seconds (200.66 M allocations: 7.502 GiB, 45.09% gc time)
(a = [-293, -178, 134, 177, -853, 511, 985, -720, 486, 181 … -946, -165, 517, 532, 915, 960, -971, 286, 486, 177], b = [0.7311473794219386, 0.3818453188579001, 0.4120712143452663, 0.2838680067429056, 0.76032556294911, 0.9119393018450566, 0.7070032863226784, 0.43326972678995523, 0.5482263849758218, 0.2625750813381811 … 0.16431670006855525, 0.37412489965948725, 0.9720419165033913, 0.08060734012468784, 0.5533171336827942, 0.9539225757859043, 0.19101056560985374, 0.9093868620405196, 0.5482263849758218, 0.2838680067429056])
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.