JuliaGenAI / JuliaGenAI/HuggingFaceDatasets.jl
Revisit format/transform interface and implementation (datasets 4 lazy Columns)
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 38
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Summary
Revisit the format / transform interface (and its implementation) now that
datasets >= 4 returns lazy Column objects for column access. The current
behavior is intentional and documented — this issue is a placeholder to
reconsider the design, not a bug report.
Current model
A Dataset has two independent underlying slots:
- the python format
pyds._format_type(set viadatasets' own
set_format):nothing,"numpy","torch","pandas","arrow",
"polars", … - the julia
jltransformfield:py2jl,identity, or a user function.
The public API couples them into a single "format" slot (src/dataset.jl,
set_format!):
"julia"≡(_format_type = nothing, jltransform = py2jl)- any python format
X≡(_format_type = X, jltransform = identity)
So the two slots are mutually exclusive through the API: setting "julia"
resets the python format to nothing; setting a python format installs
identity. The one combination that cannot be expressed is "python format X
and py2jl on top."
What prompted this
With datasets >= 4, ds["col"] returns a lazy datasets.Column, which we now
wrap in a lazy HuggingFaceDatasets.Column <: AbstractVector (added alongside
the datasets >= 4 support). It was observed that the "julia" format calls reset_format(),
so ds.pyds["col"] is always a Column.
Worth recording: resetting to nothing is not what forces the Column. In
datasets 4, Dataset.__getitem__(str) returns a Column for every format
except arrow/pandas/polars:
| python format | ds["col"] returns |
|---|---|
nothing (our "julia") |
datasets.Column (lazy) |
"numpy" |
datasets.Column (lazy) |
"torch" |
datasets.Column (lazy) |
"pandas" |
pandas.Series (native) |
"arrow" |
pyarrow.ChunkedArray (native) |
"polars" |
polars.Series (native) |
So decoupling the two formats would not avoid Columns in general — the lazy
Column is simply the v4 default for column access.
Points to reconsider
-
One format slot vs. two independent axes. Conceptually the python format
and the julia conversion are orthogonal (whatdatasetsproduces vs. a
julia-side transform). Exposing both is more faithful but roughly doubles the
state space; most combinations are redundant (py2jlalready absorbs numpy
copylessly via DLPack) or ill-defined (py2jlhas no branch for a pandas
Series/ arrow table / torch tensor). Current stance: keep the single slot;
with_jltransform/MLUtils.mapobsare the intended composition points. -
The one real limitation — numeric throughput. Under
nothing, a numeric
batch comes back as a pythonlistandpy2jlcopies it element-by-element;
under"numpy"it would be annp.ndarray→numpy2jl(zero-copy). This is
the single case where"numpy" + py2jlgenuinely wins. -
Possible internal optimization (benchmark first). Rather than exposing a
second format knob, consider whether the"julia"format should sit on
"numpy"instead ofnothing, so numeric columns transfer copyless. Tradeoff:
image columns currently arrive asPIL.Image(→Gray/RGBImageCore
views) undernothing; under"numpy"they'd be raw transposed arrays and we
lose that clean path. Best validated with theperf/harness on both numeric
and image datasets before changing anything.
Non-goals / decision so far
- Keep the current format behavior as-is for now (intentional, documented in the
guide's "The"julia"format and transforms" section). - No API change until the numeric zero-copy question (2/3) is benchmarked and we
decide whether the win justifies either an internal base-format change or a
second, opt-in axis.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/dataset.jl and the guide section on “The "julia" format and transforms” to understand the current single-slot behavior. Run the perf/ harness on both numeric and image datasets, comparing the current format with a numpy-based alternative. Done means recording benchmark results and using them to decide whether an internal optimization or an opt-in second format axis is justified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia, numpy, python
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100