HDFGroup / HDFGroup/hsds

Slow performance in arrayUtil functions with variable length data

Open
#188 0 comments 0 reactions 1 assignee Claimed by @mattjala View on GitHub
performance
Dominant language
Python
Stars
157
Forks
60
Avg merge
1d 5h
Merged PRs (30d)
15

Description

The functions that deal with converting numpy arrays of variable length elements to a buffer and back can be quite slow.
Running the test program: https://github.com/HDFGroup/hsds/blob/master/tests/perf/arrayperf/bytes_to_array.py with a million element array gave this output:

```
$ python bytes_to_array.py
getByteArraySize - elapsed: 0.3334 for 1000000 elements, returned 7888327
arrayToBytes - elapsed: 3.1166 for 1000000 elements
bytesToArray - elapsed: 1.1793
```

Not surprising since it's iterating over each element in a loop.

Looked into using numba, but numba doesn't work with numpy arrays of object type.
Cython version of arrayUtil?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.