cloudflare / cloudflare/cloudflared

Tunnel stops working

Open
#593 1 comment 0 reactions 0 assignees View on GitHub
Needs clarification Type: Bug
Dominant language
Go
Stars
15.6k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
The tunnel randomly stops working

**Environment and versions**
Docker image cloudflare/cloudflared:2022.2.0

**Logs and errors**

```
2022-03-04T08:21:22Z INF Connection cfa57569-7451-44ef-aab4-5ee99096d6df registered connIndex=2 location=IAD
2022-03-04T08:38:11Z INF Lost connection with the edge connIndex=1
2022-03-04T08:38:11Z WRN Serve tunnel error error="connection with edge closed" connIndex=1
2022-03-04T08:38:11Z INF Retrying connection in up to 1s seconds connIndex=1
2022-03-04T08:38:11Z INF Unregistered tunnel connection connIndex=1
2022-03-04T08:38:11Z INF Lost connection with the edge connIndex=3
2022-03-04T08:38:11Z WRN Serve tunnel error error="connection with edge closed" connIndex=3
2022-03-04T08:38:11Z INF Retrying connection in up to 1s seconds connIndex=3
2022-03-04T08:38:11Z INF Unregistered tunnel connection connIndex=3 2022-03-04T08:38:11Z INF Connection ac97da84-de5a-48b1-8adb-1aa61bac2386 registered connIndex=1 location=IAD
2022-03-04T08:38:11Z INF Lost connection with the edge connIndex=2
2022-03-04T08:38:11Z WRN Serve tunnel error error="connection with edge closed" connIndex=2
2022-03-04T08:38:11Z INF Retrying connection in up to 1s seconds connIndex=2
2022-03-04T08:38:11Z INF Unregistered tunnel connection connIndex=2
Recover from http2 response writer panic, error goroutine 943226123 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/cloudflare/cloudflared/connection.(*http2RespWriter).Write.func1()
/go/src/github.com/cloudflare/cloudflared/connection/http2.go:236 +0x2f
panic({0xe8c700, 0x11bd190})
/usr/local/go/src/runtime/panic.go:1038 +0x215
golang.org/x/net/http2.(*responseWriter).write(0x11c9460, 0xc001493d10, {0xc000bc5000, 0x7ff67c061a98, 0x0}, {0x0, 0x7ff6a2eb4f18})
/go/src/github.com/cloudflare/cloudflared/vendor/golang.org/x/net/http2/server.go:2708 +0x150
golang.org/x/net/http2.(*responseWriter).Write(0x0, {0xc000bc5000, 0xadb1e7, 0xc000707dc8})
/go/src/github.com/cloudflare/cloudflared/vendor/golang.org/x/net/http2/server.go:2697 +0x30
github.com/cloudflare/cloudflared/connection.(*http2RespWriter).Write(0xc000cf9380, {0xc000bc5000, 0x0, 0xdf})
/go/src/github.com/cloudflare/cloudflared/connection/http2.go:239 +0x64
zombiezen.com/go/capnproto2/rpc.(*streamTransport).SendMessage(0xc001493ce0, {0x11e4498, 0xc000cf93c0}, {{0xc000a1df50, 0x8, {0x8, 0x1}, 0xffffffffffffffff, 0x0}})
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/transport.go:65 +0x149
github.com/cloudflare/cloudflared/tunnelrpc.(*transport).SendMessage(0xc000e8adc8, {0x11e4498, 0xc000cf93c0}, {{0xc000a1df50, 0x8, {0x8, 0x1}, 0xffffffffffffffff, 0x0}}) /go/src/github.com/cloudflare/cloudflared/tunnelrpc/logtransport.go:28 +0x138 zombiezen.com/go/capnproto2/rpc.(*Conn).dispatchSend(0xc000cff600)
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/transport.go:104 +0x1b3
created by zombiezen.com/go/capnproto2/rpc.NewConn
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/rpc.go:115 +0x29d
Recover from http2 response writer panic, error goroutine 943226123 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/cloudflare/cloudflared/connection.(*http2RespWriter).Write.func1()
/go/src/github.com/cloudflare/cloudflared/connection/http2.go:236 +0x2f
panic({0xe8c700, 0x11bd190})
/usr/local/go/src/runtime/panic.go:1038 +0x215
golang.org/x/net/http2.(*responseWriter).write(0x11c9460, 0xc001493d10, {0xc000bc5000, 0x7ff67aee25d0, 0x0}, {0x0, 0x7ff6a2eb4f18})
/go/src/github.com/cloudflare/cloudflared/vendor/golang.org/x/net/http2/server.go:2708 +0x150
golang.org/x/net/http2.(*responseWriter).Write(0x0, {0xc000bc5000, 0xadb1e7, 0xc000707dc8})
/go/src/github.com/cloudflare/cloudflared/vendor/golang.org/x/net/http2/server.go:2697 +0x30
github.com/cloudflare/cloudflared/connection.(*http2RespWriter).Write(0xc000cf9380, {0xc000bc5000, 0x0, 0x35})
/go/src/github.com/cloudflare/cloudflared/connection/http2.go:239 +0x64
zombiezen.com/go/capnproto2/rpc.(*streamTransport).SendMessage(0xc001493ce0, {0x11e4498, 0xc000cf93c0}, {{0xc00c794060, 0x8, {0x8, 0x1}, 0xffffffffffffffff, 0x0}})
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/transport.go:65 +0x149
github.com/cloudflare/cloudflared/tunnelrpc.(*transport).SendMessage(0xc000e8adc8, {0x11e4498, 0xc000cf93c0}, {{0xc00c794060, 0x8, {0x8, 0x1}, 0xffffffffffffffff, 0x0}})
/go/src/github.com/cloudflare/cloudflared/tunnelrpc/logtransport.go:28 +0x138
zombiezen.com/go/capnproto2/rpc.(*Conn).dispatchSend(0xc000cff600)
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/transport.go:104 +0x1b3
created by zombiezen.com/go/capnproto2/rpc.NewConn
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/rpc.go:115 +0x29d
2022-03-04T08:38:11Z INF Lost connection with the edge connIndex=0
2022-03-04T08:38:11Z WRN Serve tunnel error error="connection with edge closed" connIndex=0
2022-03-04T08:38:11Z INF Retrying connection in up to 1s seconds connIndex=0
2022-03-04T08:38:11Z INF Unregistered tunnel connection connIndex=0
2022-03-04T08:38:12Z INF Connection 2a244dc7-b8e0-4342-a75f-9117c6130b23 registered connIndex=3 location=IAD
2022-03-04T08:38:12Z INF Connection f616308f-6ca3-43e8-a599-c5ce3dea3f5a registered connIndex=2 location=IAD
2022-03-04T08:38:12Z INF Connection 7c2787a1-f967-4d7e-875c-f27d39974d5f registered connIndex=0 location=IAD
2022-03-04T08:38:31Z INF Lost connection with the edge connIndex=3
2022-03-04T08:38:31Z WRN Serve tunnel error error="connection with edge closed" connIndex=3
2022-03-04T08:38:31Z INF Retrying connection in up to 1s seconds connIndex=3
2022-03-04T08:38:31Z INF Unregistered tunnel connection connIndex=3
2022-03-04T08:38:32Z INF Connection 686d1572-ddd6-4316-9b41-ee6ce6d54e6b registered connIndex=3 location=IAD
2022-03-04T08:40:01Z INF Lost connection with the edge connIndex=3
2022-03-04T08:40:01Z WRN Serve tunnel error error="connection with edge closed" connIndex=3
2022-03-04T08:40:01Z INF Retrying connection in up to 1s seconds connIndex=3
```

**Additional context**
Running on Kubernetes 1.19.11
Docker version 20.10.7, build 20.10.7-0ubuntu5~18.04.3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.