sqlalchemy / sqlalchemy/dogpile.cache

Concurrent usage of dbm causes cache to go stale

Open
#104 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
299
Forks
50
PR merge metrics
No merged PRs in 30d

Description

Migrated issue, originally created by David Gardner ()

Ran into this issue in production where users were reporting items in the cache being an hour old for a cache region configured with a one minute expiration time. Problem first observed with version 0.5.7, and reproduced with version 0.6.1.

I was able to reproduce the issue in a simple test of a function that returns datetime.now() (dogpile-test.py), if I run about 10 concurrent instances of the script after about a minute or so they will start reporting stale data.

However I noticed as a work-around if I use the MutexLock class from the lock_factory documentation, I don't run into the problem:
http://dogpilecache.readthedocs.io/en/latest/api.html?highlight=dogpile.cache.dbm#dogpile.cache.backends.file.DBMBackend.params.lock_factory


Attachments: dogpile-test.py

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached dogpile-test.py reproduction and the DBMBackend lock_factory documentation, then compare concurrent DBM usage with the MutexLock workaround. Reproduce the stale values under concurrent access and identify the expected expiration behavior; done means the DBM cache no longer serves expired data in that scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.