mlcommons / mlcommons/inference

[GNN] Acc eval script change: incompatibility or corrupted file?

Open
#2,119 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.6k
Forks
650
Avg merge
1d 22h
Merged PRs (30d)
6

Description

After the change to avoid using memmap when loading the labels #2081, we're encountering pickling errors via numpy.
We're unable to load the labels with numpy.load. The previous loading via np.memmap (https://github.com/mlcommons/inference/commit/be6ff52235b74a2f4ef85bf86fa3785045229fa8) did work on our environments without issues

Steps to reproduce:

  1. wget -c https://igb-public.s3.us-east-2.amazonaws.com/IGBH/processed/paper/node_label_2K.npy
  2. python -c "import numpy as np; labels = np.load('node_label_2K.npy', mmap_mode=None)"
  3. Error: ValueError: Cannot load file containing pickled data when allow_pickle=False

After setting allow_pickle=True, we face another error: _pickle.UnpicklingError: invalid load key, '\x00'

Packages used: numpy==1.26.4, torch==2.1.0+cpu

Curious if there's a dependency issue on my side.
cc: @arjunsuresh @nv-alicheng

Was the previous mode of loading the labels incorrect leading to incorrect labels? Or the current modification fixes a previously missed bug? If the latest change doesn't concern correctness of loaded labels, can we bring back the old way so as the default, and switch to np.load if --no-memmap is passed?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the wget and Python commands in the issue, then inspect the label-loading change from #2081 and commit be6ff52235b74a2f4ef85bf86fa3785045229fa8. Compare the downloaded node_label_2K.npy format with the loading modes used by the GNN accuracy evaluation. Done means identifying whether the file or loader is incompatible and establishing a verified, documented loading behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.