dwavesystems / dwavesystems/dimod

Be able to construct quadratic models from `np.memmap` array efficiently

Open
#1,312 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
143
Forks
91
Avg merge
1h 24m
Merged PRs (30d)
3

Description

**Application**
If I have a square matrix that is very large and being stored as an `np.memmap` array and I try to construct a BQM with it I often run out of memory _even if_ the actual final BQM isn't too large for my system.

**Proposed Solution**
If I simply iterate through my memory mapped matrix I don't run out of memory for matricies which are large enough that I normally would (tested up to 25,000x25,0000 on 15GB of RAM):

```{python}
bqm = dimod.BQM(vartype="BINARY")
bqm.add_variables_from({i: large_matrix[i,i] for i in range(large_matrix.shape[0])})
chunksize = 100
for row_index in range(0, large_matrix.shape[0], chunksize):
for col_index in range(0, large_matrix.shape[1], chunksize):
row_index_max = min(row_index + chunksize, large_matrix.shape[0])
col_index_max = min(col_index + chunksize, large_matrix.shape[1])
chunk = large_matrix[row_index:row_index_max, col_index:col_index_max]
quad_biases = [(i[0], i[1], j) for i, j in np.ndenumerate(chunk) if i[0] > i[1]]
bqm.add_interactions_from(quad_biases)
```
This sort of points towards a solution, but I'm not sure given how the BQM constructor is written that there is an easy way of doing this - at the moment it is cythonized and I think for that to continue to work there is likely an implicit loading of the whole memory mapped matrix into RAM which is an issue for this approach.

**Alternatives Considered**

There is the possibility of having models that are memory serialized themselves, so that you don't have to ever iterate through the matrix if you pass the underlying file, but that seems even more complex.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.