apache / apache/arrow-rs

[Proposal] Donate arrow-gpu (a cross platform gpu implementation of arrow using wgpu) to arrow-rs.

Open
#7,618 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

Hello ,

Will the team be interested in taking over the project https://github.com/psvri/arrow-gpu . This is a gpu implementation of apache arrow written in rust using [wgpu](https://github.com/gfx-rs/wgpu) .Honestly I am not able to spend more time on it due to personal life and at the same time I dont want to make the project go to waste. My hope is, if its moved to the apache foundation there will be a lot more people who would contribute and make it successful. Its around 12k lines of rust code and another few k lines of wgsl shaders.

I have implemented quite a lot of kernels and datatypes . For a list of supported kernels and dataypes you can check this [doc](https://psvri.github.io/arrow-gpu/) .

At this moment its not as performant as I would like it be, but its certainly way faster than CPU kernels. For e.g to incr 10 million non null f32 arrow arrays on my laptop takes 25% of the time compared to the cpu arrow-rs implementation (This was my first time writing gpu kernels , so perhaps someone more experienced can improve it further). Benchmark code is available [here](https://github.com/psvri/arrow-gpu/blob/main/crates/benchmarks/benches/compare_gpu_arrow.rs) .
```
gpu f32 time: [2.4419 ms 2.4573 ms 2.4732 ms]
cpu f32 time: [9.7053 ms 9.7598 ms 9.8199 ms]
```
The primary reason I had created this project was to get a cross platform, vendor independent gpu version of arrow. After researching and tinkering I landed on wgpu since thats the best lowest common denominator across platforms and vendors, and wgpu has very good chance of not being unmaintained given that its gonna be in the web .

If time had permitted I would have liked to integrate this in other projects like pyarrow, datafusion, polars and apache-spark and even aligned it with https://arrow.apache.org/docs/format/CDeviceDataInterface.html .

I am willing to help with all the efforts needed for the movement.

Sincerely,
psvri

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the arrow-gpu repository, its supported kernels and datatypes documentation, and the linked benchmark. Then assess the scope of transferring the project to arrow-rs, including its Rust code, WGSL shaders, wgpu dependency, and possible integration with Arrow projects; done would require an agreed ownership and integration plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
computer-graphics, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.