Reduce converter memory usage with on-demand read
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.4k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Required skills: Python
Difficulty: Medium
The new converter (#1151) uses member objects to operate on data from the .dat files. While these provide the necessary functions for conversion, they are also significantly larger than the original dat file entries (20x - 400x the size). This results in a comparatively high memory usage, especially for DE2 which uses ~1.1 GB of memory for all dat file entries. Note that the converter does not create objects for all entries by default, but the size difference compared to the source is still significant.
A solution to this problem could be that we read certain structures into member objects on-demand during conversion, instead of all converting them during the initial read.
This would require:
- A special
ValueMemberclass that references an offset in the .dat file and aGenieStructurereference for the dataformat. - Manual loading and unloading of the structure from file
- Dynamic loading and unloading of the structure from file
- Identifying places in the converter where manual loading makes sense
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the new converter introduced in issue #1151 and trace how .dat entries become member objects during the initial read. Identify the converter locations where manual loading could be replaced with on-demand loading and unloading. Done means the converter uses a ValueMember-style reference for eligible structures while preserving conversion behavior and reducing memory usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100