NCAS-CMS / NCAS-CMS/pyfive

numpy arrays read from attributes are not writable

Open
#53 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
104
Forks
28
Avg merge
22h 47m
Merged PRs (30d)
6

Description

I'm trying to wrap my head around the following issue. Arrays read from Datasets are created with numpy.memmap with "copyonwrite" flag set. So these numpy arrays can be changed in memory but not in the file. This is not true for arrays read from attributes. There we get back a numpy.ndarray with the writable-flag set to FALSE. An MCVE is attached below.

I was already looking where the flag might be set and why, but didn't come far. Any hints much appreciated.

MCVE:

import numpy as np
import h5py
import pyfive

with h5py.File("test.h5", mode="w") as f:
    f.attrs["att"] = np.arange(10)
    f["arr"] = np.arange(10)

with pyfive.File("test.h5") as f:
    # dataset
    arr = f["arr"][:]
    print(type(arr))
    print(arr.flags)
    print(arr.base)
    arr += 1
    print(arr)
    # attribute
    att = f.attrs["att"]
    print(type(att))
    print(att.flags)
    print(att.base)
    att += 1
    print(att)
<class 'numpy.memmap'>
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

[0 1 2 3 4 5 6 7 8 9]
[ 1  2  3  4  5  6  7  8  9 10]
<class 'numpy.ndarray'>
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : False
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

[0 1 2 3 4 5 6 7 8 9]

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [23], in <cell line: 9>()
     20 print(att.flags)
     21 print(att.base)
---> 22 att += 1
     23 print(att)

ValueError: output array is read-only

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attribute access path reached by f.attrs["att"] and compare it with the dataset read path used by f["arr"][:]. Run the MCVE against the repository to inspect the differing array flags; done means attribute arrays can be modified in memory consistently with dataset arrays and the behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.