networkdb may try to send over-sized UDP packets

GetBroadcasts in delegate.go gets the extra messages to include in cluster-wide gossip messages from two queues:

func (d *delegate) GetBroadcasts(overhead, limit int) [][]byte {
	msgs := d.nDB.networkBroadcasts.GetBroadcasts(overhead, limit)
	msgs = append(msgs, d.nDB.nodeBroadcasts.GetBroadcasts(overhead, limit)...)
	return msgs
}

However, it passes the same limit to both sub-queues.

I noticed this while reading over the code. I have not observed it in a real deployment. It's actually quite tricky to make it send an oversized packet because we have compression turned on, so you need to go quite a long way over the limit before the compressed version is too big.

With compression turned off, I was able to get it to send a 1501 byte UDP packet when the limit was 1400 (the default), with this hacked-up unit test:

func TestNetworkDBSimple(t *testing.T) {
	nodes := 10
	dbs := createNetworkDBInstances(t, nodes, "node", DefaultConfig())
	log.Printf("Created %d DBs", len(dbs))
	wg := sync.WaitGroup{}
	wg.Add(nodes)
	for i := 0; i < nodes; i++ {
		go func(i int) {
			log.Printf("Init %d", i)
			err := dbs[i].sendNodeEvent(NodeEventTypeJoin)
			if err != nil { panic(err) }
			go func() { dbs[i].sendNodeEvent(NodeEventTypeJoin) } ()
			go func() { dbs[i].sendNodeEvent(NodeEventTypeLeave) } ()
			go func() { dbs[i].sendNodeEvent(NodeEventTypeJoin) } ()
			err = dbs[i].JoinNetwork("network-a" + strconv.Itoa(i))
			if err != nil { panic(err) }
			go func() { dbs[i].JoinNetwork("network-b" + strconv.Itoa(i)) } ()
			go func() { dbs[i].JoinNetwork("network-c" + strconv.Itoa(i)) } ()
			go func() { dbs[i].JoinNetwork("network-d" + strconv.Itoa(i)) } ()
			go func() { dbs[i].JoinNetwork("network-e" + strconv.Itoa(i)) } ()
			log.Printf("Done!")
			wg.Done()
		}(i)
	}
	log.Printf("Waiting for %d nodes", nodes)
	wg.Wait()
	log.Printf("Sleeping...")
	time.Sleep(10 * time.Second)
	log.Printf("Shutting down...")
	closeNetworkDBInstances(dbs)
}

There isn't any error displayed in this case. I had to add some logging to memberlist to see it.

Start in delegate.go at delegate.GetBroadcasts and inspect how networkBroadcasts and nodeBroadcasts obtain messages. Use the described TestNetworkDBSimple scenario as a regression case, with compression disabled if needed to expose the oversized UDP packet. Done means the combined broadcasts stay within the requested limit.