Displacement Memory Growth & Purge Option + Disk Caching of Displacements
- Dominant language
- Python
- Stars
- 743
- Forks
- 150
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
Hi again,
I've been running a very large number of load combos in a distributed system, with an early stopping on failure routine. I am having trouble predicting the size of task to allocate by memory since the memory grows as I continue to perform further analysis on load combos.
I am currently addressing this with the following function running after I've processed the combo:
```python
def purge_combo(struct:'Frame3D',combo:str):
struct._D.pop(combo,None)
for node in struct.Nodes.values():
node.DX.pop(combo,None)
node.DY.pop(combo,None)
node.DZ.pop(combo,None)
node.RX.pop(combo,None)
node.RY.pop(combo,None)
node.RZ.pop(combo,None)
```
This is working well and keeping my memory consumption per task consistent (and allowing me to fully utilize my AWS resources!)
A reorganization of some of the data structures to have one authoritative dictionary for displacements `_D` would be useful, and the node dictionaries `DX,DY...RX` could be replaced with [weakref.WeakRefValueDictionary](https://docs.python.org/3/library/weakref.html#weakref.WeakValueDictionary) so that removal of the object from `_D` would automatically drop the reference from the node's as well without looping.
Of course this brings into question what happens if you want to reactivate the combo, and believe there might be some interesting options with https://github.com/grantjenks/python-diskcache although this invokes issues with how to identify the `structure,combo` pair.
Contributor guide
Research direction
The issue identifies Frame3D._D and node displacement dictionaries DX, DY, DZ, RX, RY, and RZ, plus a purge_combo function; start by locating those structures and tracing how displacement data is created, accessed, and reactivated. Define and test the intended purge behavior, combo reactivation, and any disk-cache identity scheme before choosing an implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100