apache / apache/arrow-julia

How to adapt `Arrow.Table` columns (naturally per record batch basis) into `CuArray`s for GPU processing?

Open
#294 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

I'm figuring out ways to have table columns processed by GPU, those coming from "arrow file" format files `mmap`ed for zero-copy.

The full series can not fit into GRAM, while each batch record can, so one thing is to get each contiguous chunk of the series data for upload to GPU memory in batching fashion. I've filed #293 to sort this out.

Another issue is arrow seems have intrinsic nullability, how can I adapt the array from arrow's data type to plain `float64` series for uploading to GPU?

One more issue is with Julia's lacking of nanosecond precision timestamp by default, GPU can process those as `int64`, then how can I adapt arrow nanosecond precision ts to plain `int64` arrays?

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named. Start by reading issue #293, then investigate the Arrow.Table record-batch path and CuArray upload boundary described here, including nullable columns and nanosecond timestamps; done should be a clearly documented way to handle these conversions for GPU processing.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.