AcademySoftwareFoundation / AcademySoftwareFoundation/rez

osbsolete memcache resolve when packages added / removed from local release dir

Open
#446 3 comments 1 reaction 0 assignees View on GitHub
bug cache resolver
Dominant language
Python
Stars
1.1k
Forks
369
Avg merge
12d 3h
Merged PRs (30d)
5

Description

Recently had a situation where I was retrieving an incorrect / out of date resolve from memcache, and I tracked down the situation that caused to to occur as being the following:

The resolve you are doing is for two packages: `[foo, bar]`

First `foo-2.0` is released (locally), but with only one variant, `[!bar, python-2.7]`:

```
LOCAL_RELEASE_DIR/foo/2.0 - [!bar, python-2.7]
```

Then `foo-2.0` is released (globally), but with two variants:

```
GLOBAL_RELEASE_DIR/foo/2.0 - [!bar, python-2.7]
GLOBAL_RELEASE_DIR/foo/2.0 - [bar, python-2.7]
```

Then `foo-1.9.1` (a patch release) is made globally, with two variants:

```
GLOBAL_RELEASE_DIR/foo/1.9.1 - [!bar, python-2.7]
GLOBAL_RELEASE_DIR/foo/1.9.1 - [bar, python-2.7]
```

You then do a resolve for `[foo, bar]`. Since your local release directory takes precedence, the resolve finds only ONE variant for `foo-2.0`, which is NOT compatible with `bar` - `[!bar, python-2.7]` - the resolve falls back to using `foo-1.9.1`, and your resolve is:

```
foo-1.9.1[bar, python-2.7] (GLOBAL), bar, python-2.7
```

You then DELETE your local release of `foo-2.0`... if you do a fresh resolve of `[foo, bar]` at this point, you SHOULD get:

```
foo-2.0[bar, python-2.7] (GLOBAL), bar, python-2.7
```

However, because you're using memcaching, you still end up with the OLD resolve, with `foo-1.9.1`.

What happens is:

- The key for the resolve itself doesn't include any timestamp information - just inodes for the release directories (assuming they're filesytem repos)... so the key is found, and it's associated data is then checked.
- The data retrieved for those keys DOES include timestamp info for the various packages... however, they only include timestamps for the packages in the final solve... NOT any packages which were checked / passed over in the process.
- The resolve caching process then checks if the packages themselves have been modified since the resolve, or any new releases have been made.
- Since the resolve ended up using package `foo-1.9.1`, which was the last released package, and has not been modified since the resolve, it concludes that the cache is valid.

The heart of the problems, as I implied, is that when determining whether a resolve is valid, we neither consider the timestamps of the release directories, OR the state of OTHER package files which were "checked" during the resolve attempt, but did not end up as part of the final resolve. However, both of these may affect the resolve.

Technically, the bulletproof way I can think of to ensure the cache is valid would be to store modifcation times for ALL packages checked in the process of a resolve, as well as timestamps for the release directories themselves; however, form a practical standpoint, this would greatly increase the number of filesystem "stat" calls required, and would likely decrease performance unacceptably.

A "minimum" solution would, I think, be to add timestamp information for the release directories (both top-level, and per-package-family-directory in the final solve) to the cached data (either in the key itself, or in the retrieved data). This would have prevented my particular issue, but it would still be possible to imagine scenarios where this would fail - ie, you manually deleted a variant in a local release by editing a package file by hand... something which is technically a no-no, but isn't uncommon if we're talking about local (ie, developer) release directories.

A plausible split-the-difference approach might be to store timestamps for the release directories both top-level, and per-package-family-directory in the final solve), AND timestamps for all package-files in the solve, across all release directories, not just the one in the release-directory that was "used." ie, in my example, the cached data would include the timestamp for` foo-2.0` in the LOCAL release directory, as well as `foo-2.0` in the GLOBAL directory.

Contributor guide

Open the contributing guide

Research direction

Start by tracing resolve-cache key generation and cache validation for filesystem release directories. Reproduce the local-release deletion scenario described in the issue, then determine how directory and non-selected package changes should invalidate cached results. Done means a fresh resolve returns the global foo-2.0 variant instead of the stale foo-1.9.1 result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.