influxdata / influxdata/influxdb

Possible Deadlock in TagValueIterator() and AddSeriesList()

Open
#26,164 19 comments 0 reactions 1 assignee Claimed by @devanbenz View on GitHub
1.x area/2.x area/tsi
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

**Issue Summary**
I suspect a potential deadlock related to the `TagValueIterator()` function when interacting with `AddSeriesList()`.

**Possible Deadlock Scenario**
The issue appears to arise due to conflicting RLock() and Lock() calls on f.mu within LogFile. Specifically:

1. `TagValueIterator()` acquires an `RLock()` on `f.mu`.
2. It then calls `tk.TagValueIterator()`, which attempts to acquire another `RLock()` on `tk.f.mu` (which is the same as `f.mu`).
3. Meanwhile, `AddSeriesList()` is called and attempts to acquire a write `Lock()` on `f.mu`, while `RLock()` is still held.
4. This can lead to a deadlock since Go’s `sync.RWMutex` does not allow acquiring a `Lock()` when an `RLock()` is already held.

**Relevant Code**
`TagValueIterator()` (`log_file.go`)
```go
func (f *LogFile) TagValueIterator(name, key []byte) TagValueIterator {
f.mu.RLock() // First RLock
defer f.mu.RUnlock()

mm, ok := f.mms[string(name)]
if !ok {
return nil
}

tk, ok := mm.tagSet[string(key)]
if !ok {
return nil
}
return tk.TagValueIterator() // Calls tk.TagValueIterator(), which also acquires RLock
}
```

`tk.TagValueIterator()` (`log_file.go`)
```go
func (tk *logTagKey) TagValueIterator() TagValueIterator {
tk.f.mu.RLock() // Second RLock (on the same f.mu)
a := make([]logTagValue, 0, len(tk.tagValues))
for _, v := range tk.tagValues {
a = append(a, v)
}
tk.f.mu.RUnlock()

return newLogTagValueIterator(a)
}
```

`AddSeriesList()` (`log_file.go`)
```go
func (f *LogFile) AddSeriesList(...) {
//..

f.mu.Lock() // Write lock on f.mu
defer f.mu.Unlock()

//,,.
}
```

**pprof Output When Deadlock Occurred**
```swift
goroutine 106814401 [semacquire, 6 minutes]:
sync.runtime_SemacquireMutex(0xc00015020c?, 0x78?, 0x3?)
/usr/local/go/src/runtime/sema.go:77 +0x25
sync.(*RWMutex).Lock(0xc023a48620?)
/usr/local/go/src/sync/rwmutex.go:152 +0x71
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).AddSeriesList(0xc0095a71d0, 0xc000150200, {0xc00863f800?, 0x13, 0x0?}, {0xc00863fb00?, 0x13, 0xc00e37daf8?})
influxdb-2.6.0/tsdb/index/tsi1/log_file.go:545 +0x4a5
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*Partition).createSeriesListIfNotExists(0xc037ff10e0, {0xc00863f800, 0x13, 0x20}, {0xc00863fb00, 0x13, 0x20})
influxdb-2.6.0/tsdb/index/tsi1/partition.go:725 +0x165
```
```swift
goroutine 106814631 [semacquire, 6 minutes]:
sync.runtime_SemacquireMutex(0x3318308?, 0x38?, 0xc?)
/usr/local/go/src/runtime/sema.go:77 +0x25
sync.(*RWMutex).RLock(...)
/usr/local/go/src/sync/rwmutex.go:71
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*logTagKey).TagValueIterator(0xc02a1a6fb8)
influxdb-2.6.0/tsdb/index/tsi1/log_file.go:1385 +0x51
github.com/influxdata/influxdb/v2/tsdb/index/tsi1.(*LogFile).TagValueIterator(0xc0095a71d0?, {0xc04537e640?, 0xa?, 0x158ed72?}, {0xc03be04a20, 0x9, 0x28?})
influxdb-2.6.0/tsdb/index/tsi1/log_file.go:432 +0x185
```

Currently, the only way to recover from this issue is to restart InfluxDB, which is problematic.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.