grafana / grafana/pyroscope

Data race in TestMicroServicesIntegrationV1

Open
#4,214 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

I'm almost sure it is a test-specific issue.

It's worths fixing either way:

```
==================
WARNING: DATA RACE
Write at 0x00c006755230 by goroutine 60260:
runtime.mapaccess2_faststr()
/opt/hostedtoolcache/go/1.23.8/arm64/src/runtime/map_faststr.go:117 +0x42c
github.com/grafana/dskit/ring.(*Desc).mergeWithTime()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/ring/model.go:222 +0xa9c
github.com/grafana/dskit/ring.(*Desc).Merge()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/ring/model.go:188 +0x50
github.com/grafana/dskit/kv/memberlist.computeNewValue()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/kv/memberlist/memberlist_client.go:1426 +0x50
github.com/grafana/dskit/kv/memberlist.(*KV).mergeValueForKey()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/kv/memberlist/memberlist_client.go:1377 +0x1c0
github.com/grafana/dskit/kv/memberlist.(*KV).mergeBytesValueForKey()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/kv/memberlist/memberlist_client.go:1358 +0x13c
github.com/grafana/dskit/kv/memberlist.(*KV).MergeRemoteState()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/kv/memberlist/memberlist_client.go:1319 +0x30c
github.com/hashicorp/memberlist.(*Memberlist).mergeRemoteState()
/home/runner/go/pkg/mod/github.com/grafana/memberlist@v0.3.1-0.20220708130638-bd88e10a3d91/net.go:1261 +0x190
github.com/hashicorp/memberlist.(*Memberlist).handleConn()
/home/runner/go/pkg/mod/github.com/grafana/memberlist@v0.3.1-0.20220708130638-bd88e10a3d91/net.go:309 +0xb70
github.com/hashicorp/memberlist.(*Memberlist).streamListen.gowrap1()
/home/runner/go/pkg/mod/github.com/grafana/memberlist@v0.3.1-0.20220708130638-bd88e10a3d91/net.go:224 +0x4c

Previous read at 0x00c00675[523](https://github.com/grafana/pyroscope/actions/runs/15358153111/job/43227382166?pr=4213#step:5:524)0 by goroutine 59790:
runtime.mapdelete()
/opt/hostedtoolcache/go/1.23.8/arm64/src/runtime/map.go:741 +0x43c
github.com/grafana/dskit/ring.(*Lifecycler).updateCounters()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/ring/lifecycler.go:920 +0xc8
github.com/grafana/dskit/ring.(*Lifecycler).initRing()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/ring/lifecycler.go:712 +0x618
github.com/grafana/dskit/ring.(*Lifecycler).loop()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/ring/lifecycler.go:465 +0x50
github.com/grafana/dskit/ring.(*Lifecycler).loop-fm()
:1 +0x44
github.com/grafana/dskit/services.(*BasicService).main()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:190 +0x2ac
github.com/grafana/dskit/services.(*BasicService).StartAsync.func1.gowrap1()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:119 +0x34

Goroutine 60260 (running) created at:
github.com/hashicorp/memberlist.(*Memberlist).streamListen()
/home/runner/go/pkg/mod/github.com/grafana/memberlist@v0.3.1-0.20220708130638-bd88e10a3d91/net.go:224 +0x30
github.com/hashicorp/memberlist.newMemberlist.gowrap1()
/home/runner/go/pkg/mod/github.com/grafana/memberlist@v0.3.1-0.20220708130638-bd88e10a3d91/memberlist.go:227 +0x34

Goroutine 59790 (running) created at:
github.com/grafana/dskit/services.(*BasicService).StartAsync.func1()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:119 +0x1ac
github.com/grafana/dskit/services.(*BasicService).switchState()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:139 +0xd8
github.com/grafana/dskit/services.(*BasicService).StartAsync()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:116 +0x84
github.com/grafana/dskit/ring.(*Lifecycler).StartAsync()
:1 +0x50
github.com/grafana/dskit/services.(*Manager).StartAsync()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/manager.go:79 +0xa0
github.com/grafana/dskit/services.StartManagerAndAwaitHealthy()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/manager.go:331 +0xa8
github.com/grafana/pyroscope/pkg/ingester.(*Ingester).starting()
/home/runner/work/pyroscope/pyroscope/pkg/ingester/ingester.go:152 +0x58
github.com/grafana/pyroscope/pkg/ingester.(*Ingester).starting-fm()
:1 +0x20
github.com/grafana/dskit/services.(*BasicService).main()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:157 +0xac
github.com/grafana/dskit/services.(*BasicService).StartAsync.func1.gowrap1()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:119 +0x34
==================
==================
WARNING: DATA RACE
Write at 0x00c093392c38 by goroutine 60260:
github.com/grafana/dskit/ring.(*Desc).mergeWithTime()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/ring/model.go:266 +0x934
github.com/grafana/dskit/ring.(*Desc).Merge()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/ring/model.go:188 +0x50
github.com/grafana/dskit/kv/memberlist.computeNewValue()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/kv/memberlist/memberlist_client.go:1426 +0x50
github.com/grafana/dskit/kv/memberlist.(*KV).mergeValueForKey()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/kv/memberlist/memberlist_client.go:1377 +0x1c0
github.com/grafana/dskit/kv/memberlist.(*KV).mergeBytesValueForKey()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/kv/memberlist/memberlist_client.go:1358 +0x13c
github.com/grafana/dskit/kv/memberlist.(*KV).MergeRemoteState()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/kv/memberlist/memberlist_client.go:1319 +0x30c
github.com/hashicorp/memberlist.(*Memberlist).mergeRemoteState()
/home/runner/go/pkg/mod/github.com/grafana/memberlist@v0.3.1-0.20220708130638-bd88e10a3d91/net.go:1261 +0x190
github.com/hashicorp/memberlist.(*Memberlist).handleConn()
/home/runner/go/pkg/mod/github.com/grafana/memberlist@v0.3.1-0.20220708130638-bd88e10a3d91/net.go:309 +0xb70
github.com/hashicorp/memberlist.(*Memberlist).streamListen.gowrap1()
/home/runner/go/pkg/mod/github.com/grafana/memberlist@v0.3.1-0.20220708130638-bd88e10a3d91/net.go:224 +0x4c

Previous read at 0x00c093392c38 by goroutine 59790:
github.com/grafana/dskit/ring.(*Lifecycler).updateCounters()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-2023122101[591](https://github.com/grafana/pyroscope/actions/runs/15358153111/job/43227382166?pr=4213#step:5:592)4-de83901bf4d6/ring/lifecycler.go:920 +0x98
github.com/grafana/dskit/ring.(*Lifecycler).initRing()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/ring/lifecycler.go:712 +0x618
github.com/grafana/dskit/ring.(*Lifecycler).loop()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/ring/lifecycler.go:465 +0x50
github.com/grafana/dskit/ring.(*Lifecycler).loop-fm()
:1 +0x44
github.com/grafana/dskit/services.(*BasicService).main()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:190 +0x2ac
github.com/grafana/dskit/services.(*BasicService).StartAsync.func1.gowrap1()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:119 +0x34

Goroutine 60260 (running) created at:
github.com/hashicorp/memberlist.(*Memberlist).streamListen()
/home/runner/go/pkg/mod/github.com/grafana/memberlist@v0.3.1-0.20220708130638-bd88e10a3d91/net.go:224 +0x30
github.com/hashicorp/memberlist.newMemberlist.gowrap1()
/home/runner/go/pkg/mod/github.com/grafana/memberlist@v0.3.1-0.20220708130638-bd88e10a3d91/memberlist.go:227 +0x34

Goroutine [597](https://github.com/grafana/pyroscope/actions/runs/15358153111/job/43227382166?pr=4213#step:5:598)90 (running) created at:
github.com/grafana/dskit/services.(*BasicService).StartAsync.func1()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:119 +0x1ac
github.com/grafana/dskit/services.(*BasicService).switchState()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:139 +0xd8
github.com/grafana/dskit/services.(*BasicService).StartAsync()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:116 +0x84
github.com/grafana/dskit/ring.(*Lifecycler).StartAsync()
:1 +0x50
github.com/grafana/dskit/services.(*Manager).StartAsync()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/manager.go:79 +0xa0
github.com/grafana/dskit/services.StartManagerAndAwaitHealthy()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/manager.go:331 +0xa8
github.com/grafana/pyroscope/pkg/ingester.(*Ingester).starting()
/home/runner/work/pyroscope/pyroscope/pkg/ingester/ingester.go:152 +0x58
github.com/grafana/pyroscope/pkg/ingester.(*Ingester).starting-fm()
:1 +0x20
github.com/grafana/dskit/services.(*BasicService).main()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:157 +0xac
github.com/grafana/dskit/services.(*BasicService).StartAsync.func1.gowrap1()
/home/runner/go/pkg/mod/github.com/grafana/dskit@v0.0.0-20231221015914-de83901bf4d6/services/basic_service.go:119 +0x34
==================
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating TestMicroServicesIntegrationV1 and running it with Go's race detector. Read the reported call paths through pkg/ingester/ingester.go and the dskit ring/memberlist interactions to determine the ownership conflict. Done means the integration test completes without the reported data races.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.