dotnet / dotnet/orleans

Grain location caching outlives placement-directed grain location

Open
#9,685 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

There is a bug right now where if you use a placement director to place a grain somewhere, silos make requests to that grain at that location, then later that grain is relocated and subsequently deactivated for whatever reason - the silos that cached the former location of the grain can cause the grain to be activated on an incorrect silo without making any requests for placement.

Consider the following circumstance:
There is a grain that uses a placement director to locate grains on specific silos based on criteria (e.g. alongside a client connection)

1. Client connects to silo A, triggers request to the grain, grain is placed on silo A per placement director
2. Request comes in on silo X, request is turned into a grain request which is delivered to the grain (and its location on silo A is cached)
3. Client disconnects, reconnects to silo B, the grain calls MigrateOnIdle which invokes placement director and grain is moved to silo B
4. ... time elapses, grain is deactivated for being idle
5. Request comes in on silo X, request directed to grain causes it to be activated again

Expected: Grain is not found to be active in grain directory, grain placement is calculated via placement director and grain is placed on silo B based on placement result
Actual: Grain is not found to be active in grain directory, grain is activated on silo A based on the cache held by silo X

This is a pretty challenging bug to find because at first it seems like everything is working fine - it isn't until the idle grain on silo B is deactivated and a new grain request comes in from a silo that has cached a different location.

Suggested fix:
If a request comes in to a silo from another silo due to cache, and and the grain is not active per grain directory, invoke placement before starting activation of the grain. To avoid confusing the cached location invocation with an invocation based on a placement director request, pass some flag along with the placement-directed requests that indicate this is the silo to activate the grain per placement, otherwise assume it's coming to that silo based on a cached grain directory lookup and perform the placement call on the silo receiving the request.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the grain-directory cache and placement-directed activation flow described in the reproduction scenario. Reproduce relocation from silo A to silo B, idle deactivation, and a subsequent request from a silo holding the old cache. Done means cached routing cannot reactivate the grain on the former silo, while placement-directed activation still selects the placement result.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.