[Core] GetObjectStatus never replies for an out-of-scope object whose Reference is pinned for lineage

What happened + What you expected to happen

An owner answering GetObjectStatus decides between two paths on whether it still has a Reference for the object:

https://github.com/ray-project/ray/blob/master/src/ray/core_worker/core_worker.cc#L3903-L3919

If the Reference is gone the borrower is told OUT_OF_SCOPE and raises. Otherwise the owner waits for the value with GetAsync and replies when it arrives, on the assumption stated in the comment: "The value is guaranteed to become available eventually because we own the object and its ref count is > 0."

Having a Reference does not mean the ref count is above zero. GetOwner only checks that the entry exists and carries an owner address, and lineage pinning keeps the entry well past the point where the object went out of scope. OutOfScope deliberately does not count lineage references for an object that is eligible for reconstruction, while ShouldDelete, which is what erases the entry, requires the lineage ref count to be zero:

https://github.com/ray-project/ray/blob/master/src/ray/core_worker/reference_counter.h#L360-L390

DeleteReferenceInternal acts on the first of those and not the second, so the value is deleted from the owner's memory store while the entry stays:

https://github.com/ray-project/ray/blob/master/src/ray/core_worker/reference_counter.cc#L756-L790

So for a task return that a retryable task consumed, the owner ends up holding a Reference with no value behind it. GetAsync registers the reply callback and returns, with no timeout, and only a Put takes those callbacks back out:

https://github.com/ray-project/ray/blob/master/src/ray/core_worker/store_provider/memory_store/memory_store.cc#L144-L157

Nothing puts a value back unless a task that depends on the object is retried and reconstruction runs. The two recovery entry points are both gated on the object not being out of scope, and the borrower's own Put is on the borrower's store, not the owner's. The borrower has no deadline either, since GetObjectStatus is declared with method_timeout_ms of -1.

The borrower asking at all is the case the OUT_OF_SCOPE reply was added for, described in the branch that handles it as "an edge case in the distributed ref counting protocol where a borrower dies before it can notify the owner of another borrower":

https://github.com/ray-project/ray/blob/master/src/ray/core_worker/future_resolver.cc#L58-L65

Expected: that borrower gets OUT_OF_SCOPE and raises, as it does when the Reference has already been erased. What happens instead is that its ray.get blocks, and unless a task that depends on the object is retried, no reply is ever sent. The owner has to still be alive for this: if it exits, the connection drops and the borrower gets OWNER_DIED.

Versions / Dependencies

master (167681c2b2). lineage_pinning_enabled defaults to true.

Reproduction script

The state on the owner is ordinary:

x = f.remote()        # a task return, so eligible for reconstruction
y = g.remote(x)       # g is retryable, so it holds a lineage ref on x
del x                 # x is out of scope: its value is deleted, its Reference is not

Getting a borrower to ask about x at that point needs the protocol race above, which is the part I have not staged end to end. At unit level the state is reachable directly:

reference_counter_->AddOwnedObject(object_id, {}, owner_address, "", 0,
                                   LineageReconstructionEligibility::ELIGIBLE,
                                   /*add_local_ref=*/true);
memory_store_->Put(*value, object_id, true);
reference_counter_->UpdateSubmittedTaskReferences({consumer_return_id}, {object_id});
reference_counter_->UpdateFinishedTaskReferences({consumer_return_id}, {object_id},
                                                /*release_lineage=*/false, {}, {}, &deleted);
reference_counter_->RemoveLocalReference(object_id, &deleted);
memory_store_->Delete(deleted);   // what CoreWorker::RemoveLocalReference does
// HasReference is true, the value is gone, and HandleGetObjectStatus never replies.

Note that CoreWorkerTest builds its ReferenceCounter with lineage pinning off, so the state cannot be built in that fixture as it stands.

Issue Severity

Medium: the owner-side state is ordinary, and the reply that the protocol already has for it is replaced by an indefinite block rather than an error.

AI assistance

AI assistance was used to investigate and write this up. The code references were verified against master.

Start at CoreWorker::HandleGetObjectStatus in src/ray/core_worker/core_worker.cc, then trace ReferenceCounter and memory-store behavior in reference_counter.cc, reference_counter.h, and memory_store.cc. Check how CoreWorkerTest constructs ReferenceCounter and whether lineage pinning can be enabled to reproduce the described state. Done means the borrower receives OUT_OF_SCOPE rather than waiting indefinitely when the reference remains but its value has been deleted.