ml-explore / ml-explore/mlx-data

Segfualt with prefetching and MLX arrays in key transform

Open
#47 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
483
Forks
62
PR merge metrics
No merged PRs in 30d

Description

The following code segfaults on my machine (M1 Max, OS 14.2)

Some observations:

  • Using NumPy in place of MLX works fine
  • Only segfaults with prefetching
import mlx.core as mx
from mlx.data.datasets import load_cifar10

def get_cifar10(batch_size, root=None):
    tr = load_cifar10(root=root)

    mean = mx.array([0.485, 0.456, 0.406]).reshape((1, 1, 3))
    std = mx.array([0.229, 0.224, 0.225]).reshape((1, 1, 3))

    def normalize(x):
        x = x.astype("float32") / 255.0
        return (x - mean) / std

    tr_iter = (
        tr.shuffle()
        .to_stream()
        .image_random_h_flip("image", prob=0.5)
        .pad("image", 0, 4, 4, 0.0)
        .pad("image", 1, 4, 4, 0.0)
        .image_random_crop("image", 32, 32)
        .key_transform("image", normalize)
        .batch(batch_size)
        .prefetch(4, 4)
    )

    return tr_iter

if __name__ == "__main__":
    tr_iter = get_cifar10(256)
    for batch_counter, batch in enumerate(tr_iter):
        print(batch)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied Python reproducer on the reported M1 Max and compare the MLX-array path with the NumPy path, keeping prefetching enabled. Trace the interaction between key_transform and prefetch until the native crash is isolated; done means the MLX path no longer segfaults with prefetching and the reproducer completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.