Shopify / Shopify/toxiproxy

Deadlock on writing last buffer to dead output

Open
#428 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
12.4k
Forks
510
PR merge metrics
No merged PRs in 30d

Description

It happen that during removing the Toxic deadlock appeared.

Log message on deadlock:

toxiproxy         | time="2019-12-06T21:01:03Z" level="warning" msg="Destination terminated" err=readfrom tcp 172.19.0.7:19092->172.19.0.8:60432: write tcp 172.19.0.7:19092->172.19.0.8:60432: write: broken pipe name="kafka-1" bytes=4423219

Toxiproxy v2.4.0:

		interrupted := false
		stopped := false
		for !interrupted {
			select {
			case interrupted = <-stop:
				stopped = true
			case tmp := <-link.stubs[i].Input:
				if tmp == nil {
					link.stubs[i].Close()
					if !stopped {
						<-stop
					}
					return
				}
                          ->    link.stubs[i].Output <- tmp
			}
		}

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Go forwarding loop shown in the report and trace how stop, Input, Output, and Stub.Close interact when a Toxic is removed after the destination breaks. Reproduce the deadlock using the supplied broken-pipe scenario, then verify that the removal path terminates without blocking and is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.