numpy / numpy/numpy

ENH: Add compresslevel input parameter in savez_compressed() function

Open
#20,995 11 comments 8 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
32.8k
Forks
12.8k
Avg merge
1d 7h
Merged PRs (30d)
197

Description

Proposed new feature or change:

I'm using bunch of large 2D arrays (about 10000 x 10000 ) and I need to compress and save data quickly.
The problem is that the savez_compressed() function uses only default compress level of zipfile.ZipFile class.
(The default compress level of zipfile.ZIP_DEFLATED is '6')

If I could use compress level 1 for savez_compressed() function

  • w/ Compress level 1: Compression ratio: 66.8%, exec. time: about 3.2 seconds 
  • w/ Compress level 6: Compression ratio: 69.5%, exec. time: about 24.7 seconds
    --> The compress level 1 is the most suitable for my project.
    Therefore I want to configure the compression level in savez_compressed() function.

Here is my solution. I could solve it with adding just few lines.

my modification code snippet of _savez() function definition of the numpy/lib/npyio.py file.

def _savez(file, args, kwds, compress, allow_pickle=True, pickle_kwargs=None):
    # Import is postponed to here since zipfile depends on gzip, an optional
    # component of the so-called standard library.
    import zipfile

    if not hasattr(file, 'write'):
        file = os_fspath(file)
        if not file.endswith('.npz'):
            file = file + '.npz'

    namedict = kwds
    for i, val in enumerate(args):
        key = 'arr_%d' % i
        if key in namedict.keys():
            raise ValueError(
                "Cannot use un-named variables and keyword %s" % key)
        namedict[key] = val

    if compress:
        compression = zipfile.ZIP_DEFLATED
    else:
        compression = zipfile.ZIP_STORED

    if 'compresslevel' in namedict:
        compresslevel = namedict['compresslevel']
        if not isinstance(compresslevel, int) or compresslevel < 1 or compresslevel > 9:
            compresslevel = None
    else:
        compresslevel = None

    zipf = zipfile_factory(file, mode="w", compression=compression, compresslevel=compresslevel)

    for key, val in namedict.items():
        fname = key + '.npy'
        val = np.asanyarray(val)
        # always force zip64, gh-10776
        with zipf.open(fname, 'w', force_zip64=True) as fid:
            format.write_array(fid, val,
                               allow_pickle=allow_pickle,
                               pickle_kwargs=pickle_kwargs)

    zipf.close()

It works!!

I haven't understand the github process from my idea to the merge in this kind of big open source repository cause I have no experience of github upstream activity yet. So... just let me share my idea through an issue.

thanks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with numpy/lib/npyio.py, especially the _savez() implementation and the savez_compressed() entry point, then inspect how zipfile_factory receives compression options. Compare the existing behavior with the requested configurable compression level and determine where the public API should expose it. Done means callers can select a compression level without changing the saved array contents.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.