NVIDIA / NVIDIA/DALI

Loading multiple numpy arrays in npz files (or similar)

Open
#2,953 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
C++
Stars
5.8k
Forks
678
Avg merge
3d 1h
Merged PRs (30d)
27

Description

I have structured data saved into multiple arrays in npz files. In my specific use case, one data point is actually composed of many different numpy arrays of different dtypes (int64 and float32) and "labels" are composed of numpy arrays of dtype float32, so all of this is stored in each npz file.

I understand DALI is mostly geared towards CV and audio, but from the docs I gather there is also some support for generic processing of tensors, however, I haven't been able to find any straightforward way to load this kind of data into DALI and process it in a pipeline.

Is this functionality or something similar currently suported in DALI or should I implement a custom python / C++ operator for loading data in this format (if it is at all possible)?

If something like this is not currently supported I would propose to add native support in the form of some more generic reader, since this would greatly increase the applicability of the pipelines for general problems besides CV / audio.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points. Start by reviewing DALI's existing readers and generic tensor-processing documentation, then determine whether NPZ inputs with multiple arrays and dtypes are supported; done means documenting the supported path or defining the scope of a native reader or custom-operator approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data-engineering, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.