trixi-framework / trixi-framework/Trixi.jl

Caching uses too much memory

Open
#1,825 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug low-priority testing
Dominant language
Julia
Stars
731
Forks
167
Avg merge
2d 18h
Merged PRs (30d)
25

Description

In `TrixiShallowWater.jl` I noticed error messages from `julia-actions/cache@v1` that old caches could not be deleted. This seems to be solved by setting some permissions explicitly as suggested in https://github.com/julia-actions/cache.

I was wondering if this should be also done in `Trixi.jl` or why this is not frequently encountered in `Trixi.jl`. Looking at the CI runs there I saw that this happens rarely as [here](https://github.com/trixi-framework/Trixi.jl/actions/runs/7732344766/job/21082071822#step:23:7), but most of the times it says "No existing caches found".

The reason for this is probably that our caches get deleted by github because we use too much memory. Looking at the current cache status, a single test job saves a cache of around 1GB. If I am correct it means that even a single CI run creates around 20GB of caches, so twice our cache limit.
![image](https://github.com/trixi-framework/Trixi.jl/assets/114223904/14a26fc7-ae7e-471f-b0a8-34d658ec0056)

Github will then start to delete old caches until it reaches the 10GB limit. Besides that, from what I have seen all CI jobs can still load a cache (probably also because the limit is not enforced so strictly), so it doesn't seem to cause problems right now.

I still think it would be good to keep an eye on this and find some strategies to reduce our generated cache data.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the CI cache configuration and cache-saving steps associated with the linked GitHub Actions run, then compare them with julia-actions/cache@v1 and the TrixiShallowWater.jl setup. Measure how many caches are produced and their approximate sizes. Done requires an agreed strategy that reduces generated cache data or explains why the current behavior is acceptable.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, julia
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.