How to adapt `Arrow.Table` columns (naturally per record batch basis) into `CuArray`s for GPU processing?
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
I'm figuring out ways to have table columns processed by GPU, those coming from "arrow file" format files `mmap`ed for zero-copy.
The full series can not fit into GRAM, while each batch record can, so one thing is to get each contiguous chunk of the series data for upload to GPU memory in batching fashion. I've filed #293 to sort this out.
Another issue is arrow seems have intrinsic nullability, how can I adapt the array from arrow's data type to plain `float64` series for uploading to GPU?
One more issue is with Julia's lacking of nanosecond precision timestamp by default, GPU can process those as `int64`, then how can I adapt arrow nanosecond precision ts to plain `int64` arrays?
Contributor guide
No contributing guide indexed for this repository
Research direction
No file or test is named. Start by reading issue #293, then investigate the Arrow.Table record-batch path and CuArray upload boundary described here, including nullable columns and nanosecond timestamps; done should be a clearly documented way to handle these conversions for GPU processing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100