SFTtech / SFTtech/openage

Reduce converter memory usage with on-demand read

Open
#1,256 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

area: assets good first issue lang: python
Dominant language
Python
Stars
14.4k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Required skills: Python

Difficulty: Medium

The new converter (#1151) uses member objects to operate on data from the .dat files. While these provide the necessary functions for conversion, they are also significantly larger than the original dat file entries (20x - 400x the size). This results in a comparatively high memory usage, especially for DE2 which uses ~1.1 GB of memory for all dat file entries. Note that the converter does not create objects for all entries by default, but the size difference compared to the source is still significant.

A solution to this problem could be that we read certain structures into member objects on-demand during conversion, instead of all converting them during the initial read.

This would require:

  • A special ValueMember class that references an offset in the .dat file and a GenieStructure reference for the dataformat.
  • Manual loading and unloading of the structure from file
  • Dynamic loading and unloading of the structure from file
  • Identifying places in the converter where manual loading makes sense

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the new converter introduced in issue #1151 and trace how .dat entries become member objects during the initial read. Identify the converter locations where manual loading could be replaced with on-demand loading and unloading. Done means the converter uses a ValueMember-style reference for eligible structures while preserving conversion behavior and reducing memory usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, tooling
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.