JuliaGenAI / JuliaGenAI/HuggingFaceDatasets.jl

Revisit format/transform interface and implementation (datasets 4 lazy Columns)

Open
#48 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
38
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Summary

Revisit the format / transform interface (and its implementation) now that
datasets >= 4 returns lazy Column objects for column access. The current
behavior is intentional and documented — this issue is a placeholder to
reconsider the design, not a bug report.

Current model

A Dataset has two independent underlying slots:

  • the python format pyds._format_type (set via datasets' own
    set_format): nothing, "numpy", "torch", "pandas", "arrow",
    "polars", …
  • the julia jltransform field: py2jl, identity, or a user function.

The public API couples them into a single "format" slot (src/dataset.jl,
set_format!):

  • "julia"(_format_type = nothing, jltransform = py2jl)
  • any python format X(_format_type = X, jltransform = identity)

So the two slots are mutually exclusive through the API: setting "julia"
resets the python format to nothing; setting a python format installs
identity. The one combination that cannot be expressed is "python format X
and py2jl on top."

What prompted this

With datasets >= 4, ds["col"] returns a lazy datasets.Column, which we now
wrap in a lazy HuggingFaceDatasets.Column <: AbstractVector (added alongside
the datasets >= 4 support). It was observed that the "julia" format calls reset_format(),
so ds.pyds["col"] is always a Column.

Worth recording: resetting to nothing is not what forces the Column. In
datasets 4, Dataset.__getitem__(str) returns a Column for every format
except arrow/pandas/polars:

python format ds["col"] returns
nothing (our "julia") datasets.Column (lazy)
"numpy" datasets.Column (lazy)
"torch" datasets.Column (lazy)
"pandas" pandas.Series (native)
"arrow" pyarrow.ChunkedArray (native)
"polars" polars.Series (native)

So decoupling the two formats would not avoid Columns in general — the lazy
Column is simply the v4 default for column access.

Points to reconsider

  1. One format slot vs. two independent axes. Conceptually the python format
    and the julia conversion are orthogonal (what datasets produces vs. a
    julia-side transform). Exposing both is more faithful but roughly doubles the
    state space; most combinations are redundant (py2jl already absorbs numpy
    copylessly via DLPack) or ill-defined (py2jl has no branch for a pandas
    Series / arrow table / torch tensor). Current stance: keep the single slot;
    with_jltransform / MLUtils.mapobs are the intended composition points.

  2. The one real limitation — numeric throughput. Under nothing, a numeric
    batch comes back as a python list and py2jl copies it element-by-element;
    under "numpy" it would be an np.ndarraynumpy2jl (zero-copy). This is
    the single case where "numpy" + py2jl genuinely wins.

  3. Possible internal optimization (benchmark first). Rather than exposing a
    second format knob, consider whether the "julia" format should sit on
    "numpy" instead of nothing, so numeric columns transfer copyless. Tradeoff:
    image columns currently arrive as PIL.Image (→ Gray/RGB ImageCore
    views) under nothing; under "numpy" they'd be raw transposed arrays and we
    lose that clean path. Best validated with the perf/ harness on both numeric
    and image datasets before changing anything.

Non-goals / decision so far

  • Keep the current format behavior as-is for now (intentional, documented in the
    guide's "The "julia" format and transforms" section).
  • No API change until the numeric zero-copy question (2/3) is benchmarked and we
    decide whether the win justifies either an internal base-format change or a
    second, opt-in axis.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/dataset.jl and the guide section on “The "julia" format and transforms” to understand the current single-slot behavior. Run the perf/ harness on both numeric and image datasets, comparing the current format with a numpy-based alternative. Done means recording benchmark results and using them to decide whether an internal optimization or an opt-in second format axis is justified.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia, numpy, python
Domain
data
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.