High quantity of packets will cause gopacket to miss packet.

# Table of Contents

1. [problem](#orgc493231)
2. [how to reproduce](#orgfc8dbb6)
3. [other information](#org00376d3)

# problem

I use gopacket to capture MySQL packets generated from a MySQL benchmark tool
\`mysqlslap\`. I noticed that with a high quantity of MySQL packages, even from a
single connection, will cause gopacket capture less packets than tcpdump does.
Some packets are missed with no error reported.

# how to reproduce

Use the following command to generate lots of MySQL queries:

mysqlslap --user=myuser --password=verysecretpass --host=solidcomputer --port=3306 --auto-generate-sql --concurrency=1 --number-of-queries 10000 --iterations=1

use gopacket and tcpdump to capture those queries and compare them.

When I use `number-of-queries 10000`, gopacket will print less packets than
tcpdump. Some packets's TCP sequence number can be found in tcpdump's result
but not in gopacket's result.

When I change `number-of-queries` to 500, gopacket will never miss any packet.

I tested with the default libpcap provided by Debian(version 1.8.1-3) and with the
following information:

Example code

```
package main

import (
"bufio"
"flag"
"io"
"log"
"net/http"
"time"

"github.com/google/gopacket"
"github.com/google/gopacket/examples/util"
"github.com/google/gopacket/layers"
"github.com/google/gopacket/pcap"
"github.com/google/gopacket/tcpassembly"
"github.com/google/gopacket/tcpassembly/tcpreader"
)

var iface = flag.String("i", "eth0", "Interface to get packets from")
var fname = flag.String("r", "", "Filename to read from, overrides -i")
var snaplen = flag.Int("s", 65535, "SnapLen for pcap packet capture")
var filter = flag.String("f", "tcp and port 3306", "BPF filter for pcap")
var logAllPackets = flag.Bool("v", false, "Logs every packet in great detail")

// httpStreamFactory implements tcpassembly.StreamFactory
type httpStreamFactory struct{}

// httpStream will handle the actual decoding of http requests.
type httpStream struct {
net, transport gopacket.Flow
r tcpreader.ReaderStream
}

func (h *httpStreamFactory) New(net, transport gopacket.Flow) tcpassembly.Stream {
hstream := &httpStream{
net: net,
transport: transport,
r: tcpreader.NewReaderStream(),
}
go hstream.run() // Important... we must guarantee that data from the reader stream is read.

// ReaderStream implements tcpassembly.Stream, so we can return a pointer to it.
return &hstream.r
}

func (h *httpStream) run() {
buf := bufio.NewReader(&h.r)
for {
_, err := http.ReadRequest(buf)
if err == io.EOF {
return
}
}
}

func main() {
defer util.Run()()
var handle *pcap.Handle
var err error

// Set up pcap packet capture
if *fname != "" {
log.Printf("Reading from pcap dump %q", *fname)
handle, err = pcap.OpenOffline(*fname)
} else {
log.Printf("Starting capture on interface %q", *iface)
handle, err = pcap.OpenLive(*iface, int32(*snaplen), true, pcap.BlockForever)
}
if err != nil {
log.Fatal(err)
}

if err := handle.SetBPFFilter(*filter); err != nil {
log.Fatal(err)
}

// Set up assembly
streamFactory := &httpStreamFactory{}
streamPool := tcpassembly.NewStreamPool(streamFactory)
assembler := tcpassembly.NewAssembler(streamPool)

log.Println("reading in packets")
// Read in packets, pass to assembler.
packetSource := gopacket.NewPacketSource(handle, handle.LinkType())
packets := packetSource.Packets()
ticker := time.Tick(time.Minute)
for {
select {
case packet := <-packets:
// A nil packet indicates the end of a pcap file.
if packet == nil {
return
}
if *logAllPackets {
log.Println(packet)
}
if packet.NetworkLayer() == nil || packet.TransportLayer() == nil || packet.TransportLayer().LayerType() != layers.LayerTypeTCP {
log.Println("Unusable packet")
continue
}
tcp := packet.TransportLayer().(*layers.TCP)
log.Println(tcp.Seq)
assembler.AssembleWithTimestamp(packet.NetworkLayer().NetworkFlow(), tcp, packet.Metadata().Timestamp)

case <-ticker:
// Every minute, flush connections that haven't seen activity in the past 2 minutes.
assembler.FlushOlderThan(time.Now().Add(time.Minute * -2))
}
}
}

```

System Information

```

-- System Information:
Debian Release: 9.6
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-6-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=zh_CN.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Init: systemd (via /run/systemd/system)
```

Go Information

```

go version go1.12.5 linux/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="~/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="~/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
GOROOT/bin/go version: go version go1.12.5 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.12.5
uname -sr: Linux 4.9.0-6-amd64
Distributor ID: Debian
Description: Debian GNU/Linux 9.6 (stretch)
Release: 9.6
Codename: stretch
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Debian GLIBC 2.24-11+deb9u3) stable release version 2.24, by Roland McGrath et al.

```

# other information

I tried with a minimal C code that captures packets with libpcap. It never miss
any packet, which means that the problem is not related to libpcap.

When reading from a tcpdump's pcap file instead of capture from ethernet card,
gopacket will never miss any packet whatever `number-of-queries` is set, which
means that the problem is not related to the logic gopacket handles packets(such
as tcpassembly).

I guess the problem is related to the "github.com/google/gopacket/pcap" module
but I cannot locate it.