apache / apache/arrow-julia

Precompilation / latency?

Open
#189 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

With the release 1.6, I was reading a bit about precompilation. I noticed that Arrow package has quite high latency for the first time a file / message is read. I wonder if it would be practical to generate precompile statements for the most common column types - e.g. Int/Float primitives, Strings, Dates/DateTimes + batched/dictionary encoded versions of these. Perhaps the number of methods is quite large, but it could be interesting.

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files, tests, or entry points are named. Reproduce the first-read latency described for Arrow 1.6, measure it across the listed primitive and encoded column types, and clarify the desired precompilation scope before implementation; completion criteria are not defined in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.