libp2p / libp2p/go-libp2p

peerstore: GC Purgestore races

Open
#1,712 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
6.9k
Forks
1.3k
Avg merge
13d 21h
Merged PRs (30d)
1

Description

Here some cleaner notes about a chat with @raulk about GC related stuff of addr peerstore.

#### Situation
Considering the `purgeStore()` [implementation](https://github.com/libp2p/go-libp2p-peerstore/blob/master/pstoreds/addr_book_gc.go#L256) of the GC, I found curious that no stop-the-world was happening in the underlying book. First thought, wouldn't be racy?

Since the code isn't that large, here it is to make some references later:
```go
func (gc *dsAddrBookGc) purgeStore() {
select {
case gc.running <- struct{}{}:
defer func() { <-gc.running }()
default:
// yield if lookahead is running.
return
}

record := &addrsRecord{AddrBookRecord: &pb.AddrBookRecord{}} // empty record to reuse and avoid allocs.
batch, err := newCyclicBatch(gc.ab.ds, defaultOpsPerCyclicBatch)
if err != nil {
log.Warningf("failed while creating batch to purge GC entries: %v", err)
}

results, err := gc.ab.ds.Query(purgeStoreQuery)
if err != nil {
log.Warningf("failed while opening iterator: %v", err)
return
}
defer results.Close()

// keys: /peers/addrs/
for result := range results.Next() {
record.Reset()
if err = record.Unmarshal(result.Value); err != nil {
// TODO log
continue
}

id := record.Id.ID
if !record.clean() {
continue
}

if err := record.flush(batch); err != nil {
log.Warningf("failed to flush entry modified by GC for peer: &v, err: %v", id, err)
}
gc.ab.cache.Remove(id)
}

if err = batch.Commit(); err != nil {
log.Warningf("failed to commit GC purge batch: %v", err)
}
}
```
My notes:
* `results` is a query to fetch all the records to check if they're dirty.
* `batch` is a `ds.Batch` from `ds.Batching` which I don't see has any linkage with the query. Saying it differently, no information can be inferred when flushing the batch to avoid overriding new data.
* Cache eviction (`gc.ab.cache.Remove(id)`) is done prior to flushing the batch, so even if we don't mind overriding data, it may be possible that the cache won't represent the real state of the datastore. The cache can have newer entries that were overridden by the batch. (Maybe not bad, but weird).
* Other possible scenarios for the cache are safe: i) If the batch commit fails, nothing that bad happens (only cache was wiped out). ii) the cache implementation (ARC) is thread-safe.

The naive solution would be to lock the addrbook when the GC is running, but maybe that was a conscious decision not to be made choosing performance over consistency. If that's the case, maybe useful to drop a comment line to keep that clear.

@raulk already had some thoughts about all this but recommended to create the issue to think further actions that might be worth taking.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pstoreds/addr_book_gc.go at dsAddrBookGc.purgeStore(), then inspect how the query, batch commit, and addrbook cache interact. Determine whether concurrent updates can be overwritten or leave the cache inconsistent, and whether synchronization or an explicit documented decision is needed. Done means the race and consistency behavior has been addressed or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.