dwavesystems / dwavesystems/dimod
Be able to construct quadratic models from `np.memmap` array efficiently
- Dominant language
- Python
- Stars
- 143
- Forks
- 91
- Avg merge
- 1h 24m
- Merged PRs (30d)
- 3
Description
**Application**
If I have a square matrix that is very large and being stored as an `np.memmap` array and I try to construct a BQM with it I often run out of memory _even if_ the actual final BQM isn't too large for my system.
**Proposed Solution**
If I simply iterate through my memory mapped matrix I don't run out of memory for matricies which are large enough that I normally would (tested up to 25,000x25,0000 on 15GB of RAM):
```{python}
bqm = dimod.BQM(vartype="BINARY")
bqm.add_variables_from({i: large_matrix[i,i] for i in range(large_matrix.shape[0])})
chunksize = 100
for row_index in range(0, large_matrix.shape[0], chunksize):
for col_index in range(0, large_matrix.shape[1], chunksize):
row_index_max = min(row_index + chunksize, large_matrix.shape[0])
col_index_max = min(col_index + chunksize, large_matrix.shape[1])
chunk = large_matrix[row_index:row_index_max, col_index:col_index_max]
quad_biases = [(i[0], i[1], j) for i, j in np.ndenumerate(chunk) if i[0] > i[1]]
bqm.add_interactions_from(quad_biases)
```
This sort of points towards a solution, but I'm not sure given how the BQM constructor is written that there is an easy way of doing this - at the moment it is cythonized and I think for that to continue to work there is likely an implicit loading of the whole memory mapped matrix into RAM which is an issue for this approach.
**Alternatives Considered**
There is the possibility of having models that are memory serialized themselves, so that you don't have to ever iterate through the matrix if you pass the underlying file, but that seems even more complex.
Contributor guide
Assessment
This issue has not been assessed yet.