galaxyproject / galaxyproject/galaxy

[RFC] Caching Framework

Open
#4,571 3 comments 3 reactions 0 assignees View on GitHub
area/performance friendliness/intermediate kind/feature status/planning
Dominant language
Python
Stars
1.9k
Forks
1.2k
Avg merge
2d 16h
Merged PRs (30d)
169

Description

Given recent PRs like https://github.com/galaxyproject/galaxy/pull/4495 and https://github.com/galaxyproject/galaxy/pull/4546 it seems like Galaxy might benefit from a caching framework.

I would propose:

- TTLCache, allowing things to expire even if we, in the code, fail to expire them when e.g. new tools are installed.
- A pluggable caching framework, the default being an in-memory cache object. I've used https://pypi.python.org/pypi/cachetools and been happy with it. I'll contribute a redis cache backend or something like this.
- We look at aggresively caching things that aren't changed often, e.g. the toolbox.
- We have cache keys including useful information, e.g. the toolbox might be under `tools.toolbox`, so we can decide when a new tool is installed, we should expire everything with `tools.` in it / prefixing it.

cc @mvdbeek how would you feel about this?

Contributor guide

Open the contributing guide

Research direction

Review PRs 4495 and 4546 first, then compare the proposed TTLCache, pluggable backends, toolbox caching, and prefixed cache keys with cachetools. The issue names no implementation files or tests; done would require an agreed design and implementation plan before coding can begin.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, redis
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.