JWock82 / JWock82/Pynite

Displacement Memory Growth & Purge Option + Disk Caching of Displacements

Open
#178 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
743
Forks
150
Avg merge
6d 1h
Merged PRs (30d)
2

Description

Hi again,

I've been running a very large number of load combos in a distributed system, with an early stopping on failure routine. I am having trouble predicting the size of task to allocate by memory since the memory grows as I continue to perform further analysis on load combos.

I am currently addressing this with the following function running after I've processed the combo:

```python
def purge_combo(struct:'Frame3D',combo:str):

struct._D.pop(combo,None)
for node in struct.Nodes.values():
node.DX.pop(combo,None)
node.DY.pop(combo,None)
node.DZ.pop(combo,None)
node.RX.pop(combo,None)
node.RY.pop(combo,None)
node.RZ.pop(combo,None)
```
This is working well and keeping my memory consumption per task consistent (and allowing me to fully utilize my AWS resources!)

A reorganization of some of the data structures to have one authoritative dictionary for displacements `_D` would be useful, and the node dictionaries `DX,DY...RX` could be replaced with [weakref.WeakRefValueDictionary](https://docs.python.org/3/library/weakref.html#weakref.WeakValueDictionary) so that removal of the object from `_D` would automatically drop the reference from the node's as well without looping.

Of course this brings into question what happens if you want to reactivate the combo, and believe there might be some interesting options with https://github.com/grantjenks/python-diskcache although this invokes issues with how to identify the `structure,combo` pair.

Contributor guide

Open the contributing guide

Research direction

The issue identifies Frame3D._D and node displacement dictionaries DX, DY, DZ, RX, RY, and RZ, plus a purge_combo function; start by locating those structures and tracing how displacement data is created, accessed, and reactivated. Define and test the intended purge behavior, combo reactivation, and any disk-cache identity scheme before choosing an implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.