Change processing of EFS data to xarray
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 148
- Forks
- 55
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 18
Description
Feature Request
With the introduction of ReadEfs in PR #202, more EFS data is retained for sector coupling studies. The issue is, if many sectors are using the EFS data and try to process it at the same time, (I think) memory consumption skyrockets as I am sometimes (and seemingly randomly) getting killed errors when running. This has not been an issue for electricity only studies or if running the script by itself.
Suggested Solution
I believe xarray will be much more memory efficient at processing EFS data.
Additional Info
No response
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the ReadEfs changes from PR #202 and trace how retained EFS data is processed when multiple sectors run together. Reproduce the reported memory growth or killed errors in a sector-coupling study, then evaluate whether xarray can replace the current processing approach while preserving the EFS data needed for those studies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100