algorand / algorand/go-algorand
Broken test: TestPartitionRecoverySwapStartup
- Dominant language
- Go
- Stars
- 1.4k
- Forks
- 537
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 17
Description
## Description
Travis Test Errors Reference Spreadsheet for details on cases where this test has failed: https://docs.google.com/spreadsheets/d/1apjaNvZodnTSIdZaQlpMaqf0nzbKgSYdIWHqvmINHS8/edit#gid=0
### Example Failure
Time: Thu 7/15/2021
Build: https://app.travis-ci.com/github/algorand/go-algorand/jobs/524585138#L1951
Travis Test: Ubuntu AMD64 Integration Test, MacOS AMD64 Integration Test
Package: test/e2e-go/features/partitionRecovery
Test: TestPartitionRecoverySwapStartup
```
=== FAIL: test/e2e-go/features/partitionRecovery TestPartitionRecoverySwapStartup (unknown)
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionRecoverySwapStartup/Wallet1.rootkey
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionRecoverySwapStartup/Wallet2.rootkey
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionRecoverySwapStartup/Wallet2.0.3000000.partkey
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionRecoverySwapStartup/Wallet1.0.3000000.partkey
https://github.com/[secure]foundation/specs/tree/65b4ab3266c52c56a0fa7d591754887d68faad0a 100000
algod(11933) : No REST API Token found. Generated token: f710125cc41731ac3477d269f489851f3f4e16805f5daa2e6d5ad03c9fcfc55e
algod(11933) : No Admin REST API Token found. Generated token: 5549ec3be1cbbd91648616c93b1fca2e9a8fe914c086bbb280fa06146042dc49
algod(11933) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionRecoverySwapStartup/Relay/node.log
algod(11933) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11933) : Initializing the Algorand node...
algod(11933) : Success!
algod(11933) : ⇨ http server started on 127.0.0.1:35433
algod(11933) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:35433. Press Ctrl-C to exit
algod(11946) : No REST API Token found. Generated token: 4525bcbcaa6638d5522c4ccd65e9a5c784864e5761643cbe00745ac46652faee
algod(11946) : No Admin REST API Token found. Generated token: 5d453bc17d022d4401efc808b83f4e1edfe1cb8691a99e165908b87ac85b0f22
algod(11946) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionRecoverySwapStartup/Node1/node.log
algod(11946) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11946) : Initializing the Algorand node...
algod(11946) : Success!
algod(11946) : ⇨ http server started on 127.0.0.1:40007
algod(11946) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:40007. Press Ctrl-C to exit
algod(11961) : No REST API Token found. Generated token: 9292ea05b39c6d1d5fb2fbf65b7b9275a6c56af8288fa5bc30e5187c88f75555
algod(11961) : No Admin REST API Token found. Generated token: 013fdefefb1af28b98882b44f4e7eba63610beb9dc64d2cb8496c30743b9a76e
algod(11961) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionRecoverySwapStartup/Node2/node.log
algod(11961) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11961) : Initializing the Algorand node...
algod(11961) : Success!
algod(11961) : ⇨ http server started on 127.0.0.1:45915
algod(11961) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:45915. Press Ctrl-C to exit
algod(11820) : goroutine 170 [running]:
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod.captureCallstack(0x4a73f4, 0x10, 0x1000000000000d0)
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/deadlockLogger.go:59 +0x7f
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod.(*deadlockLogger).onPotentialDeadlock.func1()
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/deadlockLogger.go:73 +0x5e
algod(11820) : sync.(*Once).doSlow(0xc00007bc28, 0xc0039c9890)
algod(11820) : sync/once.go:66 +0x104
algod(11820) : sync.(*Once).Do(0xc00007bc28, 0xc0039c9890)
algod(11820) : sync/once.go:57 +0x69
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod.(*deadlockLogger).onPotentialDeadlock(0xc00007bc00)
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/deadlockLogger.go:71 +0x62
algod(11820) : github.com/[secure]/go-deadlock.lock(0xc002fcbe70, 0x20a2520, 0xc000225680, 0x0)
algod(11820) : github.com/[secure]/go-deadlock@v0.2.1/deadlock.go:252 +0xa59
algod(11820) : github.com/[secure]/go-deadlock.(*Mutex).Lock(0xc000225680)
algod(11820) : github.com/[secure]/go-deadlock@v0.2.1/deadlock.go:80 +0xec
algod(11820) : github.com/[secure]/go-[secure]/data.(*AccountManager).Keys(0xc000225680, 0x2, 0x0, 0x0, 0x0)
algod(11820) : github.com/[secure]/go-[secure]/data/accountManager.go:71 +0x6d
algod(11820) : github.com/[secure]/go-[secure]/node.(*AlgorandFullNode).VotingKeys(0xc000386500, 0x2, 0x0, 0x2328, 0x222e0, 0x2710)
algod(11820) : github.com/[secure]/go-[secure]/node/node.go:1114 +0x82
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*asyncPseudonode).loadRoundParticipationKeys(0xc002f0efa0, 0x2, 0xc00476f7a0, 0xc000321c80, 0xc002dd10d0)
algod(11820) : github.com/[secure]/go-[secure]/agreement/pseudonode.go:218 +0x3bd
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*pseudonodeBaseTask).populateParticipationKeys(...)
algod(11820) : github.com/[secure]/go-[secure]/agreement/pseudonode.go:347
algod(11820) : github.com/[secure]/go-[secure]/agreement.asyncPseudonode.makeVotesTask(0x20a22c0, 0xc000386500, 0x20a3e60, 0xc0000b05c0, 0x20a22e0, 0xc000386500, 0x20d5560, 0xc0000b05a0, 0x20ea0c0, 0xc0001b2a60, ...)
algod(11820) : github.com/[secure]/go-[secure]/agreement/pseudonode.go:252 +0x1a9
algod(11820) : github.com/[secure]/go-[secure]/agreement.asyncPseudonode.MakeVotes(0x20a22c0, 0xc000386500, 0x20a3e60, 0xc0000b05c0, 0x20a22e0, 0xc000386500, 0x20d5560, 0xc0000b05a0, 0x20ea0c0, 0xc0001b2a60, ...)
algod(11820) : github.com/[secure]/go-[secure]/agreement/pseudonode.go:182 +0x16d
algod(11820) : github.com/[secure]/go-[secure]/agreement.pseudonodeAction.do(0xc, 0x2, 0x0, 0x1, 0x0, 0xffcb4d6322878a00, 0x62f691370d78e598, 0x2bc75af1447bc2fb, 0x9f8d595ae63837a6, 0xc48748b28dbe8ebb, ...)
algod(11820) : github.com/[secure]/go-[secure]/agreement/actions.go:376 +0xda3
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*Service).do(0xc0002a7900, 0x20bf780, 0xc000258640, 0xc002fcbb50, 0x1, 0x1)
algod(11820) : github.com/[secure]/go-[secure]/agreement/service.go:255 +0x98
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*Service).demuxLoop(0xc0002a7900, 0x20bf780, 0xc000258640, 0xc000351f80, 0xc0000920c0, 0xc000092120)
algod(11820) : github.com/[secure]/go-[secure]/agreement/service.go:163 +0x110
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.(*Service).Start
algod(11820) : github.com/[secure]/go-[secure]/agreement/service.go:146 +0x756
algod(11820) : goroutine 1 [select, 1 minutes]:
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod.(*Server).Start(0xc0048915a0)
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/server.go:235 +0x1195
algod(11820) : main.run(0x0)
algod(11820) : github.com/[secure]/go-[secure]/cmd/algod/main.go:365 +0x1ca2
algod(11820) : main.main()
algod(11820) : github.com/[secure]/go-[secure]/cmd/algod/main.go:61 +0xc4
algod(11820) : goroutine 445 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*wsPeer).writeLoop(0xc002de8000)
algod(11820) : github.com/[secure]/go-[secure]/network/wsPeer.go:643 +0x331
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*wsPeer).init
algod(11820) : github.com/[secure]/go-[secure]/network/wsPeer.go:356 +0x5c6
algod(11820) : goroutine 12743 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0xc, 0xc0007c3880, 0xc0007a1b90, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 409 [IO wait]:
algod(11820) : internal/poll.runtime_pollWait(0x7fd68e080f48, 0x72, 0x20a4b20)
algod(11820) : runtime/netpoll.go:203 +0x55
algod(11820) : internal/poll.(*pollDesc).wait(0xc000283698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
algod(11820) : internal/poll/fd_poll_runtime.go:87 +0xe4
algod(11820) : internal/poll.(*pollDesc).waitRead(...)
algod(11820) : internal/poll/fd_poll_runtime.go:92
algod(11820) : internal/poll.(*FD).Read(0xc000283680, 0xc0003ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
algod(11820) : internal/poll/fd_unix.go:169 +0x253
algod(11820) : net.(*netFD).Read(0xc000283680, 0xc0003ab000, 0x1000, 0x1000, 0x7fd6676eefff, 0x400, 0x7fd6676b3e00)
algod(11820) : net/fd_unix.go:202 +0x66
algod(11820) : net.(*conn).Read(0xc0000105c0, 0xc0003ab000, 0x1000, 0x1000, 0x1a0010000000000, 0x0, 0xc002f076a0)
algod(11820) : net/net.go:184 +0xa1
algod(11820) : bufio.(*Reader).fill(0xc00026e360)
algod(11820) : bufio/bufio.go:100 +0x19a
algod(11820) : bufio.(*Reader).Peek(0xc00026e360, 0x2, 0xc002f077f8, 0x44e03a, 0xc000001800, 0xc0000b0020, 0x3)
algod(11820) : bufio/bufio.go:138 +0xa0
algod(11820) : github.com/[secure]/websocket.(*Conn).read(0xc000c00000, 0x2, 0xc002f07878, 0x5295ba, 0xc0038900a8, 0xc00007cf01, 0x417fea)
algod(11820) : github.com/[secure]/websocket@v1.4.2/conn.go:422 +0x6d
algod(11820) : github.com/[secure]/websocket.(*Conn).advanceFrame(0xc000c00000, 0x0, 0x0, 0x3755140)
algod(11820) : github.com/[secure]/websocket@v1.4.2/conn.go:888 +0x8d
algod(11820) : github.com/[secure]/websocket.(*Conn).NextReader(0xc000c00000, 0xc002f07bc8, 0x2, 0x0, 0xc003876300, 0x2)
algod(11820) : github.com/[secure]/websocket@v1.4.2/conn.go:1059 +0x10b
algod(11820) : github.com/[secure]/go-[secure]/network.(*wsPeer).readLoop(0xc000287d40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsPeer.go:388 +0x248
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*wsPeer).init
algod(11820) : github.com/[secure]/go-[secure]/network/wsPeer.go:355 +0x5a1
algod(11820) : goroutine 12680 [semacquire]:
algod(11820) : sync.runtime_SemacquireMutex(0xc000386534, 0x900000000, 0x1)
algod(11820) : runtime/sema.go:71 +0x47
algod(11820) : sync.(*Mutex).lockSlow(0xc000386530)
algod(11820) : sync/mutex.go:138 +0x1c1
algod(11820) : sync.(*Mutex).Lock(0xc000386530)
algod(11820) : sync/mutex.go:81 +0x7d
algod(11820) : github.com/[secure]/go-deadlock.lock.func1(0xc0032f51e0, 0xc0032a9c20)
algod(11820) : github.com/[secure]/go-deadlock@v0.2.1/deadlock.go:205 +0x35
algod(11820) : created by github.com/[secure]/go-deadlock.lock
algod(11820) : github.com/[secure]/go-deadlock@v0.2.1/deadlock.go:204 +0x136
algod(11820) : goroutine 420 [select]:
algod(11820) : github.com/[secure]/go-deadlock.lock(0xc0033fba10, 0x20a2520, 0xc000225680, 0xc000e50400)
algod(11820) : github.com/[secure]/go-deadlock@v0.2.1/deadlock.go:211 +0x26d
algod(11820) : github.com/[secure]/go-deadlock.(*Mutex).Lock(0xc000225680)
algod(11820) : github.com/[secure]/go-deadlock@v0.2.1/deadlock.go:80 +0xec
algod(11820) : github.com/[secure]/go-[secure]/data.(*AccountManager).AddParticipation(0xc000225680, 0x2762eb313e59e198, 0x277d0cc5015a42ac, 0x3cd89b207ef92926, 0x6d2d0780e329cff7, 0xc00355d680, 0xc000e504b0, 0x0, 0x2dc6c0, 0x2710, ...)
algod(11820) : github.com/[secure]/go-[secure]/data/accountManager.go:100 +0x5e
algod(11820) : github.com/[secure]/go-[secure]/node.(*AlgorandFullNode).loadParticipationKeys(0xc000386500, 0xc00005f6f0, 0x2)
algod(11820) : github.com/[secure]/go-[secure]/node/node.go:812 +0xe41
algod(11820) : github.com/[secure]/go-[secure]/node.(*AlgorandFullNode).checkForParticipationKeys(0xc000386500)
algod(11820) : github.com/[secure]/go-[secure]/node/node.go:754 +0x1eb
algod(11820) : created by github.com/[secure]/go-[secure]/node.(*AlgorandFullNode).startMonitoringRoutines
algod(11820) : github.com/[secure]/go-[secure]/node/node.go:400 +0x71
algod(11820) : goroutine 308 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 307 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 306 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 289 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 288 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 287 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 286 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 285 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 284 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 283 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 282 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 281 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 280 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 276 [select]:
algod(11820) : github.com/[secure]/go-[secure]/compactcert.(*Worker).signer(0xc0002e3380, 0x0)
algod(11820) : github.com/[secure]/go-[secure]/compactcert/signer.go:64 +0x503
algod(11820) : created by github.com/[secure]/go-[secure]/compactcert.(*Worker).Start
algod(11820) : github.com/[secure]/go-[secure]/compactcert/worker.go:108 +0x362
algod(11820) : goroutine 279 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).meshThread(0xc0002aa300)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1522 +0x5b6
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:793 +0x4cb
algod(11820) : goroutine 277 [select]:
algod(11820) : github.com/[secure]/go-[secure]/compactcert.(*Worker).builder(0xc0002e3380, 0x0)
algod(11820) : github.com/[secure]/go-[secure]/compactcert/builder.go:201 +0x1b3
algod(11820) : created by github.com/[secure]/go-[secure]/compactcert.(*Worker).Start
algod(11820) : github.com/[secure]/go-[secure]/compactcert/worker.go:111 +0x3a8
algod(11820) : goroutine 174 [select, 1 minutes]:
algod(11820) : github.com/[secure]/go-[secure]/data.(*TxHandler).backlogWorker(0xc0003148c0)
algod(11820) : github.com/[secure]/go-[secure]/data/txHandler.go:139 +0x2d7
algod(11820) : created by github.com/[secure]/go-[secure]/data.(*TxHandler).Start
algod(11820) : github.com/[secure]/go-[secure]/data/txHandler.go:103 +0x18f
algod(11820) : goroutine 173 [select]:
algod(11820) : github.com/[secure]/go-[secure]/rpcs.(*BlockService).listenForCatchupReq(0xc000282800, 0xc000326c00, 0xc000092180)
algod(11820) : github.com/[secure]/go-[secure]/rpcs/blockService.go:245 +0x263
algod(11820) : created by github.com/[secure]/go-[secure]/rpcs.(*BlockService).Start
algod(11820) : github.com/[secure]/go-[secure]/rpcs/blockService.go:128 +0x117
algod(11820) : goroutine 172 [select]:
algod(11820) : github.com/[secure]/go-[secure]/rpcs.(*TxSyncer).Start.func1(0xc000282900, 0xc000039200)
algod(11820) : github.com/[secure]/go-[secure]/rpcs/txSyncer.go:87 +0x2a6
algod(11820) : created by github.com/[secure]/go-[secure]/rpcs.(*TxSyncer).Start
algod(11820) : github.com/[secure]/go-[secure]/rpcs/txSyncer.go:79 +0x74
algod(11820) : goroutine 171 [chan send]:
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*Service).mainLoop(0xc0002a7900, 0xc000351f80, 0xc0000920c0, 0xc000092120)
algod(11820) : github.com/[secure]/go-[secure]/agreement/service.go:225 +0x80d
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.(*Service).Start
algod(11820) : github.com/[secure]/go-[secure]/agreement/service.go:147 +0x7a2
algod(11820) : goroutine 169 [select]:
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*asyncPersistenceLoop).loop(0xc000356b40, 0x20bf780, 0xc000ef6040)
algod(11820) : github.com/[secure]/go-[secure]/agreement/persistence.go:282 +0x1d8
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.(*asyncPersistenceLoop).Start
algod(11820) : github.com/[secure]/go-[secure]/agreement/persistence.go:270 +0xf5
algod(11820) : goroutine 168 [select]:
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*pseudonodeVerifier).verifierLoop(0xc0000a2d90, 0xc00013e140)
algod(11820) : github.com/[secure]/go-[secure]/agreement/pseudonode.go:329 +0x22a
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.asyncPseudonode.makePseudonodeVerifier
algod(11820) : github.com/[secure]/go-[secure]/agreement/pseudonode.go:264 +0x17a
algod(11820) : goroutine 402 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 353 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 352 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 12744 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0xd, 0xc0007c38f0, 0xc0007c3880, 0xc0039a0d80, 0x4)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 12745 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0xf, 0xc0007c39d0, 0xc0007c3960, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 236 [select, 1 minutes]:
algod(11820) : database/sql.(*DB).connectionOpener(0xc0002e2e40, 0x20bf780, 0xc00007a780)
algod(11820) : database/sql/sql.go:1071 +0x140
algod(11820) : created by database/sql.OpenDB
algod(11820) : database/sql/sql.go:742 +0x25d
algod(11820) : goroutine 238 [chan receive, 1 minutes]:
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*AsyncVoteVerifier).worker(0xc000033400)
algod(11820) : github.com/[secure]/go-[secure]/agreement/asyncVoteVerifier.go:88 +0x117
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.MakeAsyncVoteVerifier
algod(11820) : github.com/[secure]/go-[secure]/agreement/asyncVoteVerifier.go:82 +0x2cb
algod(11820) : goroutine 210 [sync.Cond.Wait]:
algod(11820) : runtime.goparkunlock(...)
algod(11820) : runtime/proc.go:310
algod(11820) : sync.runtime_notifyListWait(0xc00007a550, 0x1)
algod(11820) : runtime/sema.go:513 +0xf8
algod(11820) : sync.(*Cond).Wait(0xc00007a540)
algod(11820) : sync/cond.go:56 +0x8e
algod(11820) : github.com/[secure]/go-[secure]/ledger.(*blockNotifier).worker(0xc0001f52e8)
algod(11820) : github.com/[secure]/go-[secure]/ledger/notifier.go:55 +0xc1
algod(11820) : created by github.com/[secure]/go-[secure]/ledger.(*blockNotifier).loadFromDisk
algod(11820) : github.com/[secure]/go-[secure]/ledger/notifier.go:93 +0x162
algod(11820) : goroutine 195 [select, 1 minutes]:
algod(11820) : github.com/[secure]/go-[secure]/ledger.(*accountUpdates).commitSyncer(0xc0001f4dd8, 0xc0000b8d00)
algod(11820) : github.com/[secure]/go-[secure]/ledger/acctupdates.go:2042 +0x1b4
algod(11820) : created by github.com/[secure]/go-[secure]/ledger.(*accountUpdates).initializeFromDisk
algod(11820) : github.com/[secure]/go-[secure]/ledger/acctupdates.go:1185 +0xc29
algod(11820) : goroutine 12746 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0x11, 0xc000b2e000, 0xc0007c3f80, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 105 [select, 1 minutes]:
algod(11820) : database/sql.(*DB).connectionOpener(0xc0001360c0, 0x20bf780, 0xc0000d03c0)
algod(11820) : database/sql/sql.go:1071 +0x140
algod(11820) : created by database/sql.OpenDB
algod(11820) : database/sql/sql.go:742 +0x25d
algod(11820) : goroutine 12747 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0x10, 0xc0007c3f80, 0xc0007c39d0, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 137 [select, 1 minutes]:
algod(11820) : database/sql.(*DB).connectionOpener(0xc0002e2c00, 0x20bf780, 0xc00007a300)
algod(11820) : database/sql/sql.go:1071 +0x140
algod(11820) : created by database/sql.OpenDB
algod(11820) : database/sql/sql.go:742 +0x25d
algod(11820) : goroutine 102 [select]:
algod(11820) : github.com/[secure]/go-[secure]/util/execpool.(*backlog).worker(0xc0000c1a40)
algod(11820) : github.com/[secure]/go-[secure]/util/execpool/backlog.go:133 +0x1db
algod(11820) : created by github.com/[secure]/go-[secure]/util/execpool.MakeBacklog
algod(11820) : github.com/[secure]/go-[secure]/util/execpool/backlog.go:70 +0x28a
algod(11820) : goroutine 101 [select, 1 minutes]:
algod(11820) : github.com/[secure]/go-[secure]/util/execpool.(*backlog).worker(0xc0000c1980)
algod(11820) : github.com/[secure]/go-[secure]/util/execpool/backlog.go:133 +0x1db
algod(11820) : created by github.com/[secure]/go-[secure]/util/execpool.MakeBacklog
algod(11820) : github.com/[secure]/go-[secure]/util/execpool/backlog.go:70 +0x28a
algod(11820) : goroutine 100 [select]:
algod(11820) : github.com/[secure]/go-[secure]/util/execpool.(*pool).worker(0xc0000d1f80)
algod(11820) : github.com/[secure]/go-[secure]/util/execpool/pool.go:150 +0x301
algod(11820) : created by github.com/[secure]/go-[secure]/util/execpool.MakePool
algod(11820) : github.com/[secure]/go-[secure]/util/execpool/pool.go:85 +0x254
algod(11820) : goroutine 99 [select]:
algod(11820) : github.com/[secure]/go-[secure]/util/execpool.(*pool).worker(0xc0000d1f80)
algod(11820) : github.com/[secure]/go-[secure]/util/execpool/pool.go:150 +0x301
algod(11820) : created by github.com/[secure]/go-[secure]/util/execpool.MakePool
algod(11820) : github.com/[secure]/go-[secure]/util/execpool/pool.go:85 +0x254
algod(11820) : goroutine 135 [select, 1 minutes]:
algod(11820) : database/sql.(*DB).connectionOpener(0xc0002e20c0, 0x20bf780, 0xc00007a000)
algod(11820) : database/sql/sql.go:1071 +0x140
algod(11820) : created by database/sql.OpenDB
algod(11820) : database/sql/sql.go:742 +0x25d
algod(11820) : goroutine 136 [select, 1 minutes]:
algod(11820) : database/sql.(*DB).connectionOpener(0xc0002e2b40, 0x20bf780, 0xc00007a2c0)
algod(11820) : database/sql/sql.go:1071 +0x140
algod(11820) : created by database/sql.OpenDB
algod(11820) : database/sql/sql.go:742 +0x25d
algod(11820) : goroutine 111 [sync.Cond.Wait]:
algod(11820) : runtime.goparkunlock(...)
algod(11820) : runtime/proc.go:310
algod(11820) : sync.runtime_notifyListWait(0xc0000d1a10, 0x700000001)
algod(11820) : runtime/sema.go:513 +0xf8
algod(11820) : sync.(*Cond).Wait(0xc0000d1a00)
algod(11820) : sync/cond.go:56 +0x8e
algod(11820) : github.com/[secure]/go-[secure]/ledger.(*blockQueue).syncer(0xc000378140)
algod(11820) : github.com/[secure]/go-[secure]/ledger/blockqueue.go:99 +0xe2
algod(11820) : created by github.com/[secure]/go-[secure]/ledger.bqInit
algod(11820) : github.com/[secure]/go-[secure]/ledger/blockqueue.go:72 +0x3b3
algod(11820) : goroutine 12677 [select]:
algod(11820) : github.com/[secure]/go-deadlock.lock(0xc0032f51e0, 0x20a2520, 0xc000386528, 0x59fe00)
algod(11820) : github.com/[secure]/go-deadlock@v0.2.1/deadlock.go:211 +0x26d
algod(11820) : github.com/[secure]/go-deadlock.(*Mutex).Lock(0xc000386528)
algod(11820) : github.com/[secure]/go-deadlock@v0.2.1/deadlock.go:80 +0xec
algod(11820) : github.com/[secure]/go-[secure]/node.(*AlgorandFullNode).Status(0xc000386500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0340471330ae603, 0x70fbd935f, ...)
algod(11820) : github.com/[secure]/go-[secure]/node/node.go:660 +0x139
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/api/server/v1/handlers.getNodeStatus(0xc000386500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/api/server/v1/handlers/handlers.go:48 +0x76
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/api/server/v1/handlers.Status(0xc000386500, 0x20ea0c0, 0xc0001b2a60, 0x0, 0x0, 0xc00026b1a0, 0x20ec7a0, 0xc002f0f4a0)
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/api/server/v1/handlers/handlers.go:530 +0xe3
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/api/server.wrapCtx.func1(0x20ec7a0, 0xc002f0f4a0, 0x40, 0xc000128940)
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/api/server/router.go:89 +0x97
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/api/server/lib/middlewares.(*AuthMiddleware).handler.func1(0x20ec7a0, 0xc002f0f4a0, 0x1, 0x1)
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/api/server/lib/middlewares/auth.go:100 +0x56f
algod(11820) : github.com/labstack/echo/v4.(*Echo).add.func1(0x20ec7a0, 0xc002f0f4a0, 0x1b8868a, 0x1b)
algod(11820) : github.com/labstack/echo/v4@v4.1.17/echo.go:522 +0x7e
algod(11820) : github.com/labstack/echo/v4/middleware.CORSWithConfig.func1.1(0x20ec7a0, 0xc002f0f4a0, 0x2ccd0a0, 0xc004014580)
algod(11820) : github.com/labstack/echo/v4@v4.1.17/middleware/cors.go:121 +0x67c
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/api/server/lib/middlewares.(*LoggerMiddleware).handler.func1(0x20ec7a0, 0xc002f0f4a0, 0x2, 0x2)
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/api/server/lib/middlewares/logger.go:52 +0x11f
algod(11820) : github.com/labstack/echo/v4.(*Echo).ServeHTTP.func1(0x20ec7a0, 0xc002f0f4a0, 0x1, 0x415700)
algod(11820) : github.com/labstack/echo/v4@v4.1.17/echo.go:627 +0x260
algod(11820) : github.com/labstack/echo/v4/middleware.RemoveTrailingSlashWithConfig.func1.1(0x20ec7a0, 0xc002f0f4a0, 0x1, 0x1)
algod(11820) : github.com/labstack/echo/v4@v4.1.17/middleware/slash.go:118 +0x252
algod(11820) : github.com/labstack/echo/v4.(*Echo).ServeHTTP(0xc000c30000, 0x20baac0, 0xc0002d0460, 0xc002f98e00)
algod(11820) : github.com/labstack/echo/v4@v4.1.17/echo.go:633 +0x1ff
algod(11820) : net/http.serverHandler.ServeHTTP(0x2ccd4e0, 0x20baac0, 0xc0002d0460, 0xc002f98e00)
algod(11820) : net/http/server.go:2836 +0xcf
algod(11820) : net/http.(*conn).serve(0xc002f0f400, 0x20bf780, 0xc004230dc0)
algod(11820) : net/http/server.go:1924 +0x838
algod(11820) : created by net/http.(*Server).Serve
algod(11820) : net/http/server.go:2962 +0x5b7
algod(11820) : goroutine 239 [select, 1 minutes]:
algod(11820) : database/sql.(*DB).connectionOpener(0xc0002e3080, 0x20bf780, 0xc00007b040)
algod(11820) : database/sql/sql.go:1071 +0x140
algod(11820) : created by database/sql.OpenDB
algod(11820) : database/sql/sql.go:742 +0x25d
algod(11820) : goroutine 12678 [IO wait]:
algod(11820) : internal/poll.runtime_pollWait(0x7fd68e080ca8, 0x72, 0x20a4b20)
algod(11820) : runtime/netpoll.go:203 +0x55
algod(11820) : internal/poll.(*pollDesc).wait(0xc003dd6498, 0x72, 0x0, 0x1, 0xffffffffffffffff)
algod(11820) : internal/poll/fd_poll_runtime.go:87 +0xe4
algod(11820) : internal/poll.(*pollDesc).waitRead(...)
algod(11820) : internal/poll/fd_poll_runtime.go:92
algod(11820) : internal/poll.(*FD).Read(0xc003dd6480, 0xc00363f5a1, 0x1, 0x1, 0x0, 0x0, 0x0)
algod(11820) : internal/poll/fd_unix.go:169 +0x253
algod(11820) : net.(*netFD).Read(0xc003dd6480, 0xc00363f5a1, 0x1, 0x1, 0x20bf800, 0xc002f32420, 0x0)
algod(11820) : net/fd_unix.go:202 +0x66
algod(11820) : net.(*conn).Read(0xc003cd04b8, 0xc00363f5a1, 0x1, 0x1, 0x44046c, 0xc000330f30, 0x46f1c0)
algod(11820) : net/net.go:184 +0xa1
algod(11820) : net/http.(*connReader).backgroundRead(0xc00363f590)
algod(11820) : net/http/server.go:689 +0x92
algod(11820) : created by net/http.(*connReader).startBackgroundRead
algod(11820) : net/http/server.go:685 +0x187
algod(11820) : goroutine 351 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 541 [syscall, 1 minutes]:
algod(11820) : os/signal.signal_recv(0x472811)
algod(11820) : runtime/sigqueue.go:147 +0x9c
algod(11820) : os/signal.loop()
algod(11820) : os/signal/signal_unix.go:23 +0x30
algod(11820) : created by os/signal.Notify.func1
algod(11820) : os/signal/signal.go:127 +0x7c
algod(11820) : goroutine 247 [select, 1 minutes]:
algod(11820) : database/sql.(*DB).connectionOpener(0xc0002e3140, 0x20bf780, 0xc00007bc80)
algod(11820) : database/sql/sql.go:1071 +0x140
algod(11820) : created by database/sql.OpenDB
algod(11820) : database/sql/sql.go:742 +0x25d
algod(11820) : goroutine 12748 [select]:
algod(11820) : net/http.(*persistConn).roundTrip(0xc000c06d80, 0xc0038b47b0, 0x0, 0x0, 0x0)
algod(11820) : net/http/transport.go:2498 +0x8c2
algod(11820) : net/http.(*Transport).roundTrip(0xc000291400, 0xc0038a8700, 0xf, 0xc0340479aa5a0e01, 0x0)
algod(11820) : net/http/transport.go:565 +0xf6c
algod(11820) : net/http.(*Transport).RoundTrip(...)
algod(11820) : net/http/roundtrip.go:17
algod(11820) : github.com/[secure]/go-[secure]/network.(*rateLimitingTransport).RoundTrip(0xc0002aa8f8, 0xc0038a8700, 0xc0002aa8f8, 0x0, 0x0)
algod(11820) : github.com/[secure]/go-[secure]/network/rateLimitingTransport.go:72 +0x2c1
algod(11820) : net/http.send(0xc0038a8700, 0x20a22a0, 0xc0002aa8f8, 0x0, 0x0, 0x0, 0x457030, 0x0, 0xc000ed3470, 0x45720f)
algod(11820) : net/http/client.go:252 +0x6a4
algod(11820) : net/http.(*Client).send(0xc0039a1108, 0xc0038a8700, 0x0, 0x0, 0x0, 0xc003d81200, 0x0, 0x1, 0x80)
algod(11820) : net/http/client.go:176 +0x1cb
algod(11820) : net/http.(*Client).do(0xc0039a1108, 0xc0038a8700, 0x0, 0x0, 0x0)
algod(11820) : net/http/client.go:699 +0x2cd
algod(11820) : net/http.(*Client).Do(...)
algod(11820) : net/http/client.go:567
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*HTTPFetcher).getBlockBytes(0xc000ed3bb0, 0x20bf780, 0xc003068900, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/universalFetcher.go:223 +0x682
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*universalBlockFetcher).fetchBlock(0xc0038ba600, 0x20bf780, 0xc003068900, 0x2, 0x19e5700, 0xc0039a10e0, 0x17054e0, 0x0, 0x0, 0xc0038a2a20, ...)
algod(11820) : github.com/[secure]/go-[secure]/catchup/universalFetcher.go:76 +0x831
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).innerFetch(0xc0002db000, 0x2, 0x19e5700, 0xc0039a10e0, 0x0, 0x0, 0x0, 0x0, 0x0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:173 +0x3d6
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0x2, 0xc000785810, 0xc0007857a0, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:219 +0x15d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 269 [select]:
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*poolCryptoVerifier).voteFillWorker(0xc00028b0e0, 0xc000039440)
algod(11820) : github.com/[secure]/go-[secure]/agreement/cryptoVerifier.go:197 +0x2c1
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.makeCryptoVerifier
algod(11820) : github.com/[secure]/go-[secure]/agreement/cryptoVerifier.go:181 +0x4e4
algod(11820) : goroutine 271 [chan receive]:
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*poolCryptoVerifier).proposalVerifyWorker(0xc00028b0e0)
algod(11820) : github.com/[secure]/go-[secure]/agreement/cryptoVerifier.go:349 +0x100
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.makeCryptoVerifier
algod(11820) : github.com/[secure]/go-[secure]/agreement/cryptoVerifier.go:183 +0x532
algod(11820) : goroutine 270 [chan receive, 1 minutes]:
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*poolCryptoVerifier).bundleWaitWorker(0xc00028b0e0, 0xc000039440)
algod(11820) : github.com/[secure]/go-[secure]/agreement/cryptoVerifier.go:239 +0xd4
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.makeCryptoVerifier
algod(11820) : github.com/[secure]/go-[secure]/agreement/cryptoVerifier.go:182 +0x510
algod(11820) : goroutine 268 [chan receive]:
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*AsyncVoteVerifier).worker(0xc000033a40)
algod(11820) : github.com/[secure]/go-[secure]/agreement/asyncVoteVerifier.go:88 +0x117
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.MakeAsyncVoteVerifier
algod(11820) : github.com/[secure]/go-[secure]/agreement/asyncVoteVerifier.go:82 +0x2cb
algod(11820) : goroutine 267 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch(0xc0002db000, 0x2)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:445 +0xb94
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).sync(0xc0002db000)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:568 +0x296
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).periodicSync(0xc0002db000)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:520 +0x8b3
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).Start
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:128 +0x1bc
algod(11820) : goroutine 255 [select, 1 minutes]:
algod(11820) : database/sql.(*DB).connectionOpener(0xc0002e3200, 0x20bf780, 0xc0000d1a80)
algod(11820) : database/sql/sql.go:1071 +0x140
algod(11820) : created by database/sql.OpenDB
algod(11820) : database/sql/sql.go:742 +0x25d
algod(11820) : goroutine 165 [select]:
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*demux).tokenizeMessages.func1(0xc000351e60, 0xc0003568a0, 0xc000283180, 0x1b70fdb, 0x2, 0x1bf11e0, 0x20c1c40, 0xc000032f50, 0x20bf780, 0xc000ef6000)
algod(11820) : github.com/[secure]/go-[secure]/agreement/demux.go:117 +0x197
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.(*demux).tokenizeMessages
algod(11820) : github.com/[secure]/go-[secure]/agreement/demux.go:112 +0x11a
algod(11820) : goroutine 164 [select]:
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*demux).tokenizeMessages.func1(0xc000351e00, 0xc0003567e0, 0xc000283180, 0x1b70f67, 0x2, 0x1bf11e8, 0x20c1c40, 0xc000032f50, 0x20bf780, 0xc000ef6000)
algod(11820) : github.com/[secure]/go-[secure]/agreement/demux.go:148 +0x648
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.(*demux).tokenizeMessages
algod(11820) : github.com/[secure]/go-[secure]/agreement/demux.go:112 +0x11a
algod(11820) : goroutine 167 [select]:
algod(11820) : github.com/[secure]/go-[secure]/agreement.pseudonodeProposalsTask.execute(0xc0046e0960, 0x20bf780, 0xc000258640, 0xc00026a480, 0xc0002d02a0, 0x3, 0x3, 0x2, 0x0, 0xc000033a40, ...)
algod(11820) : github.com/[secure]/go-[secure]/agreement/pseudonode.go:532 +0x1787
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*pseudonodeVerifier).verifierLoop(0xc0000a25f0, 0xc00013e000)
algod(11820) : github.com/[secure]/go-[secure]/agreement/pseudonode.go:339 +0x111
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.asyncPseudonode.makePseudonodeVerifier
algod(11820) : github.com/[secure]/go-[secure]/agreement/pseudonode.go:264 +0x17a
algod(11820) : goroutine 264 [select, 1 minutes]:
algod(11820) : database/sql.(*DB).connectionOpener(0xc0002e32c0, 0x20bf780, 0xc0002583c0)
algod(11820) : database/sql/sql.go:1071 +0x140
algod(11820) : created by database/sql.OpenDB
algod(11820) : database/sql/sql.go:742 +0x25d
algod(11820) : goroutine 166 [select, 1 minutes]:
algod(11820) : github.com/[secure]/go-[secure]/agreement.(*demux).tokenizeMessages.func1(0xc000351ec0, 0xc000356900, 0xc000283180, 0x1b71023, 0x2, 0x1bf11d8, 0x20c1c40, 0xc000032f50, 0x20bf780, 0xc000ef6000)
algod(11820) : github.com/[secure]/go-[secure]/agreement/demux.go:117 +0x197
algod(11820) : created by github.com/[secure]/go-[secure]/agreement.(*demux).tokenizeMessages
algod(11820) : github.com/[secure]/go-[secure]/agreement/demux.go:112 +0x11a
algod(11820) : goroutine 12741 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0xb, 0xc0007a1b90, 0xc0007a0fc0, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 403 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 404 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 405 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).messageHandlerThread(0xc0002aa300, 0xc000327a40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1151 +0x23c
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:806 +0x60a
algod(11820) : goroutine 406 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).broadcastThread(0xc0002aa300)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1334 +0x7d6
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:809 +0x663
algod(11820) : goroutine 407 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).prioWeightRefresh(0xc0002aa300)
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:1738 +0x44a
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*WebsocketNetwork).Start
algod(11820) : github.com/[secure]/go-[secure]/network/wsNetwork.go:812 +0x98c
algod(11820) : goroutine 421 [select]:
algod(11820) : github.com/[secure]/go-[secure]/node.(*AlgorandFullNode).txPoolGaugeThread(0xc000386500)
algod(11820) : github.com/[secure]/go-[secure]/node/node.go:831 +0x260
algod(11820) : created by github.com/[secure]/go-[secure]/node.(*AlgorandFullNode).startMonitoringRoutines
algod(11820) : github.com/[secure]/go-[secure]/node/node.go:402 +0x93
algod(11820) : goroutine 410 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*wsPeer).writeLoop(0xc000287d40)
algod(11820) : github.com/[secure]/go-[secure]/network/wsPeer.go:643 +0x331
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*wsPeer).init
algod(11820) : github.com/[secure]/go-[secure]/network/wsPeer.go:356 +0x5c6
algod(11820) : goroutine 444 [IO wait]:
algod(11820) : internal/poll.runtime_pollWait(0x7fd68e080e68, 0x72, 0x20a4b20)
algod(11820) : runtime/netpoll.go:203 +0x55
algod(11820) : internal/poll.(*pollDesc).wait(0xc000283998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
algod(11820) : internal/poll/fd_poll_runtime.go:87 +0xe4
algod(11820) : internal/poll.(*pollDesc).waitRead(...)
algod(11820) : internal/poll/fd_poll_runtime.go:92
algod(11820) : internal/poll.(*FD).Read(0xc000283980, 0xc002dde000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
algod(11820) : internal/poll/fd_unix.go:169 +0x253
algod(11820) : net.(*netFD).Read(0xc000283980, 0xc002dde000, 0x1000, 0x1000, 0x7fd6676b5b00, 0x20300000000000, 0x7fd6676eefff)
algod(11820) : net/fd_unix.go:202 +0x66
algod(11820) : net.(*conn).Read(0xc0000105d0, 0xc002dde000, 0x1000, 0x1000, 0xc000bf1690, 0x42099e, 0x7fd68e0ed290)
algod(11820) : net/net.go:184 +0xa1
algod(11820) : bufio.(*Reader).fill(0xc000edb0e0)
algod(11820) : bufio/bufio.go:100 +0x19a
algod(11820) : bufio.(*Reader).Peek(0xc000edb0e0, 0x2, 0xc000c04300, 0xc0000b0020, 0x3, 0xc000050880, 0xc000050800)
algod(11820) : bufio/bufio.go:138 +0xa0
algod(11820) : github.com/[secure]/websocket.(*Conn).read(0xc002de6000, 0x2, 0xc003891e58, 0xc002dfda01, 0x417fea, 0xc0000a41b0, 0x417e69)
algod(11820) : github.com/[secure]/websocket@v1.4.2/conn.go:422 +0x6d
algod(11820) : github.com/[secure]/websocket.(*Conn).advanceFrame(0xc002de6000, 0x0, 0x0, 0x7fd68e8347d0)
algod(11820) : github.com/[secure]/websocket@v1.4.2/conn.go:888 +0x8d
algod(11820) : github.com/[secure]/websocket.(*Conn).NextReader(0xc002de6000, 0xc000bf1c50, 0xc003908901, 0x13f71f616f4fbe01, 0x9, 0xa)
algod(11820) : github.com/[secure]/websocket@v1.4.2/conn.go:1059 +0x10b
algod(11820) : github.com/[secure]/go-[secure]/network.(*wsPeer).readLoop(0xc002de8000)
algod(11820) : github.com/[secure]/go-[secure]/network/wsPeer.go:388 +0x248
algod(11820) : created by github.com/[secure]/go-[secure]/network.(*wsPeer).init
algod(11820) : github.com/[secure]/go-[secure]/network/wsPeer.go:355 +0x5a1
algod(11820) : goroutine 422 [syscall]:
algod(11820) : github.com/[secure]/go-[secure]/crypto._Cfunc_crypto_sign_ed25519_seed_keypair(0xc0038d2060, 0xc0038ceec0, 0xc0038d2040, 0x0)
algod(11820) : _cgo_gotypes.go:84 +0x6b
algod(11820) : github.com/[secure]/go-[secure]/crypto.ed25519GenerateKeySeed(...)
algod(11820) : github.com/[secure]/go-[secure]/crypto/curve25519.go:100
algod(11820) : github.com/[secure]/go-[secure]/crypto.ed25519GenerateKeyRNG(0x20a1fe0, 0x3753360, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
algod(11820) : github.com/[secure]/go-[secure]/crypto/curve25519.go:96 +0x214
algod(11820) : github.com/[secure]/go-[secure]/crypto.(*OneTimeSignatureSecrets).DeleteBeforeFineGrained(0xc000e503c0, 0x0, 0x2, 0x2710)
algod(11820) : github.com/[secure]/go-[secure]/crypto/onetimesig.go:395 +0x764
algod(11820) : github.com/[secure]/go-[secure]/data/account.PersistedParticipation.DeleteOldKeys(0x8d61be3ab334a38f, 0xd056cedcc9151b1c, 0x3716efd22d582d77, 0xbec9e0e3448fb855, 0xc000038d80, 0xc000e503c0, 0x0, 0x2dc6c0, 0x2710, 0xc0002e3200, ...)
algod(11820) : github.com/[secure]/go-[secure]/data/account/participation.go:137 +0x15c
algod(11820) : github.com/[secure]/go-[secure]/data.(*AccountManager).DeleteOldKeys.func1(0xc000225680, 0xc002ec0ee8, 0xc000c334d0, 0xc002ec09d0, 0xc002ec1058, 0xc002ec0870)
algod(11820) : github.com/[secure]/go-[secure]/data/accountManager.go:174 +0x587
algod(11820) : github.com/[secure]/go-[secure]/data.(*AccountManager).DeleteOldKeys(0xc000225680, 0x1, 0x6e5b34e8044e2cb1, 0xf1bf18ebb20b47e6, 0x68989461a5838d2f, 0xfedd4718ef8e5fd8, 0x95b7bc0884fda459, 0x62d38e8003d4e4f8, 0xf44a0c07bd4d9b34, 0x76ab91ed1a5a147b, ...)
algod(11820) : github.com/[secure]/go-[secure]/data/accountManager.go:178 +0x1fa
algod(11820) : github.com/[secure]/go-[secure]/node.(*AlgorandFullNode).oldKeyDeletionThread(0xc000386500)
algod(11820) : github.com/[secure]/go-[secure]/node/node.go:916 +0xaae
algod(11820) : created by github.com/[secure]/go-[secure]/node.(*AlgorandFullNode).startMonitoringRoutines
algod(11820) : github.com/[secure]/go-[secure]/node/node.go:404 +0xb5
algod(11820) : goroutine 13172 [IO wait]:
algod(11820) : internal/poll.runtime_pollWait(0x7fd68e080ae8, 0x72, 0x20a4b20)
algod(11820) : runtime/netpoll.go:203 +0x55
algod(11820) : internal/poll.(*pollDesc).wait(0xc0048c3d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
algod(11820) : internal/poll/fd_poll_runtime.go:87 +0xe4
algod(11820) : internal/poll.(*pollDesc).waitRead(...)
algod(11820) : internal/poll/fd_poll_runtime.go:92
algod(11820) : internal/poll.(*FD).Read(0xc0048c3d80, 0xc000b47000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
algod(11820) : internal/poll/fd_unix.go:169 +0x253
algod(11820) : net.(*netFD).Read(0xc0048c3d80, 0xc000b47000, 0x1000, 0x1000, 0x1, 0x20b5340, 0x37538a0)
algod(11820) : net/fd_unix.go:202 +0x66
algod(11820) : net.(*conn).Read(0xc003d80478, 0xc000b47000, 0x1000, 0x1000, 0x10, 0x10, 0xc000bb5968)
algod(11820) : net/net.go:184 +0xa1
algod(11820) : net/http.(*persistConn).Read(0xc000c07d40, 0xc000b47000, 0x1000, 0x1000, 0x0, 0xf7b, 0x1)
algod(11820) : net/http/transport.go:1825 +0xe8
algod(11820) : bufio.(*Reader).fill(0xc003273740)
algod(11820) : bufio/bufio.go:100 +0x19a
algod(11820) : bufio.(*Reader).Peek(0xc003273740, 0x1, 0x2, 0x0, 0x0, 0xc0038c6100, 0x0)
algod(11820) : bufio/bufio.go:138 +0xa0
algod(11820) : net/http.(*persistConn).readLoop(0xc000c07d40)
algod(11820) : net/http/transport.go:1978 +0x2af
algod(11820) : created by net/http.(*Transport).dialConn
algod(11820) : net/http/transport.go:1647 +0xc0f
algod(11820) : goroutine 13129 [select]:
algod(11820) : net/http.(*persistConn).writeLoop(0xc000c070e0)
algod(11820) : net/http/transport.go:2277 +0x1c3
algod(11820) : created by net/http.(*Transport).dialConn
algod(11820) : net/http/transport.go:1648 +0xc34
algod(11820) : goroutine 13128 [IO wait]:
algod(11820) : internal/poll.runtime_pollWait(0x7fd68e080a08, 0x72, 0x20a4b20)
algod(11820) : runtime/netpoll.go:203 +0x55
algod(11820) : internal/poll.(*pollDesc).wait(0xc0048c3898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
algod(11820) : internal/poll/fd_poll_runtime.go:87 +0xe4
algod(11820) : internal/poll.(*pollDesc).waitRead(...)
algod(11820) : internal/poll/fd_poll_runtime.go:92
algod(11820) : internal/poll.(*FD).Read(0xc0048c3880, 0xc002ddf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
algod(11820) : internal/poll/fd_unix.go:169 +0x253
algod(11820) : net.(*netFD).Read(0xc0048c3880, 0xc002ddf000, 0x1000, 0x1000, 0x1, 0x20b5340, 0x37538a0)
algod(11820) : net/fd_unix.go:202 +0x66
algod(11820) : net.(*conn).Read(0xc003d80430, 0xc002ddf000, 0x1000, 0x1000, 0x10, 0x10, 0xc000bb7968)
algod(11820) : net/net.go:184 +0xa1
algod(11820) : net/http.(*persistConn).Read(0xc000c070e0, 0xc002ddf000, 0x1000, 0x1000, 0x0, 0xf7b, 0x1)
algod(11820) : net/http/transport.go:1825 +0xe8
algod(11820) : bufio.(*Reader).fill(0xc0032725a0)
algod(11820) : bufio/bufio.go:100 +0x19a
algod(11820) : bufio.(*Reader).Peek(0xc0032725a0, 0x1, 0x2, 0x0, 0x0, 0xc00385e500, 0x0)
algod(11820) : bufio/bufio.go:138 +0xa0
algod(11820) : net/http.(*persistConn).readLoop(0xc000c070e0)
algod(11820) : net/http/transport.go:1978 +0x2af
algod(11820) : created by net/http.(*Transport).dialConn
algod(11820) : net/http/transport.go:1647 +0xc0f
algod(11820) : goroutine 478 [IO wait]:
algod(11820) : internal/poll.runtime_pollWait(0x7fd68e080bc8, 0x72, 0x72)
algod(11820) : runtime/netpoll.go:203 +0x55
algod(11820) : internal/poll.(*pollDesc).wait(0xc000283c98, 0x72, 0x0, 0x0, 0x1b74c2b)
algod(11820) : internal/poll/fd_poll_runtime.go:87 +0xe4
algod(11820) : internal/poll.(*pollDesc).waitRead(...)
algod(11820) : internal/poll/fd_poll_runtime.go:92
algod(11820) : internal/poll.(*FD).Accept(0xc000283c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
algod(11820) : internal/poll/fd_unix.go:384 +0x2c9
algod(11820) : net.(*netFD).accept(0xc000283c80, 0x44e03a, 0xc000bf7800, 0x2ccd580)
algod(11820) : net/fd_unix.go:238 +0x56
algod(11820) : net.(*TCPListener).accept(0xc000c1c140, 0x461d79, 0x19806e7e508, 0x4c33bb6)
algod(11820) : net/tcpsock_posix.go:139 +0x50
algod(11820) : net.(*TCPListener).Accept(0xc000c1c140, 0x47449c, 0x44812e, 0xc000c0abe8, 0x46f800)
algod(11820) : net/tcpsock.go:261 +0x50
algod(11820) : net/http.(*Server).Serve(0x2ccd4e0, 0x20ba800, 0xc000c1c140, 0x0, 0x0)
algod(11820) : net/http/server.go:2930 +0x3fe
algod(11820) : github.com/labstack/echo/v4.(*Echo).StartServer(0xc000c30000, 0x2ccd4e0, 0x7fd655920008, 0x0)
algod(11820) : github.com/labstack/echo/v4@v4.1.17/echo.go:724 +0x6fa
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod.(*Server).Start.func1(0xc000c30000, 0xc000c36900)
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/server.go:225 +0x45
algod(11820) : created by github.com/[secure]/go-[secure]/daemon/algod.(*Server).Start
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/server.go:224 +0xec7
algod(11820) : goroutine 12742 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0xe, 0xc0007c3960, 0xc0007c38f0, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 13100 [select]:
algod(11820) : net/http.(*persistConn).writeLoop(0xc000c07680)
algod(11820) : net/http/transport.go:2277 +0x1c3
algod(11820) : created by net/http.(*Transport).dialConn
algod(11820) : net/http/transport.go:1648 +0xc34
algod(11820) : goroutine 16635 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).innerFetch.func1(0xc0038c6ba0, 0xc0002db000, 0x3, 0xc0037adb70)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:167 +0x11e
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).innerFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:166 +0x387
algod(11820) : goroutine 16286 [semacquire]:
algod(11820) : sync.runtime_SemacquireMutex(0xc00022568c, 0x1100000000, 0x1)
algod(11820) : runtime/sema.go:71 +0x47
algod(11820) : sync.(*Mutex).lockSlow(0xc000225688)
algod(11820) : sync/mutex.go:138 +0x1c1
algod(11820) : sync.(*Mutex).Lock(0xc000225688)
algod(11820) : sync/mutex.go:81 +0x7d
algod(11820) : github.com/[secure]/go-deadlock.lock.func1(0xc0033fba10, 0xc0035cc2a0)
algod(11820) : github.com/[secure]/go-deadlock@v0.2.1/deadlock.go:205 +0x35
algod(11820) : created by github.com/[secure]/go-deadlock.lock
algod(11820) : github.com/[secure]/go-deadlock@v0.2.1/deadlock.go:204 +0x136
algod(11820) : goroutine 13173 [select]:
algod(11820) : net/http.(*persistConn).writeLoop(0xc000c07d40)
algod(11820) : net/http/transport.go:2277 +0x1c3
algod(11820) : created by net/http.(*Transport).dialConn
algod(11820) : net/http/transport.go:1648 +0xc34
algod(11820) : goroutine 12740 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0xa, 0xc0007a0fc0, 0xc000748700, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 12737 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0x6, 0xc000748620, 0xc0007485b0, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 12735 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0x5, 0xc0007485b0, 0xc0007858f0, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 12734 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0x4, 0xc0007858f0, 0xc000785880, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 12739 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0x9, 0xc000748700, 0xc000748690, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 12736 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0x7, 0xc0007a02a0, 0xc000748620, 0xc0039a0d80, 0xc00026e008)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 12738 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0x8, 0xc000748690, 0xc0007a02a0, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:228 +0x2943
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 16605 [select]:
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).innerFetch.func1(0xc0038c6480, 0xc0002db000, 0x2, 0xc0037ad940)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:167 +0x11e
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).innerFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:166 +0x387
algod(11820) : goroutine 13099 [IO wait]:
algod(11820) : internal/poll.runtime_pollWait(0x7fd68e080928, 0x72, 0x20a4b20)
algod(11820) : runtime/netpoll.go:203 +0x55
algod(11820) : internal/poll.(*pollDesc).wait(0xc0048c3b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
algod(11820) : internal/poll/fd_poll_runtime.go:87 +0xe4
algod(11820) : internal/poll.(*pollDesc).waitRead(...)
algod(11820) : internal/poll/fd_poll_runtime.go:92
algod(11820) : internal/poll.(*FD).Read(0xc0048c3b00, 0xc000c02000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
algod(11820) : internal/poll/fd_unix.go:169 +0x253
algod(11820) : net.(*netFD).Read(0xc0048c3b00, 0xc000c02000, 0x1000, 0x1000, 0x1, 0x20b5340, 0x37538a0)
algod(11820) : net/fd_unix.go:202 +0x66
algod(11820) : net.(*conn).Read(0xc003cd0220, 0xc000c02000, 0x1000, 0x1000, 0x10, 0x10, 0xc000c11968)
algod(11820) : net/net.go:184 +0xa1
algod(11820) : net/http.(*persistConn).Read(0xc000c07680, 0xc000c02000, 0x1000, 0x1000, 0x0, 0xf7b, 0x1)
algod(11820) : net/http/transport.go:1825 +0xe8
algod(11820) : bufio.(*Reader).fill(0xc003417e60)
algod(11820) : bufio/bufio.go:100 +0x19a
algod(11820) : bufio.(*Reader).Peek(0xc003417e60, 0x1, 0x2, 0x0, 0x0, 0xc003900800, 0x0)
algod(11820) : bufio/bufio.go:138 +0xa0
algod(11820) : net/http.(*persistConn).readLoop(0xc000c07680)
algod(11820) : net/http/transport.go:1978 +0x2af
algod(11820) : created by net/http.(*Transport).dialConn
algod(11820) : net/http/transport.go:1647 +0xc0f
algod(11820) : goroutine 12733 [select]:
algod(11820) : github.com/[secure]/go-[secure]/network.(*wsPeer).Request(0xc000287d40, 0x20bf800, 0xc0038b9260, 0x1b71015, 0x2, 0xc0038d61e0, 0x2, 0x4, 0x0, 0x0, ...)
algod(11820) : github.com/[secure]/go-[secure]/network/wsPeer.go:832 +0x909
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*wsFetcherClient).requestBlock(0xc0038b4930, 0x20bf800, 0xc0038b9260, 0x3, 0xc0038b9260, 0xc0037adbc0, 0xc003883c80, 0x40a719, 0xc000be7b78)
algod(11820) : github.com/[secure]/go-[secure]/catchup/universalFetcher.go:162 +0x36e
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*wsFetcherClient).getBlockBytes(0xc0038b4930, 0x20bf780, 0xc003068940, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/universalFetcher.go:136 +0x1ec
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*universalBlockFetcher).fetchBlock(0xc0038ba900, 0x20bf780, 0xc003068940, 0x3, 0x1b3a920, 0xc000287d40, 0x17054e0, 0x0, 0x0, 0xc0038a2ca0, ...)
algod(11820) : github.com/[secure]/go-[secure]/catchup/universalFetcher.go:66 +0x1b2
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).innerFetch(0xc0002db000, 0x3, 0x1b3a920, 0xc000287d40, 0x0, 0x0, 0x0, 0x0, 0x0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:173 +0x3d6
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).fetchAndWrite(0xc0002db000, 0x3, 0xc000785880, 0xc000785810, 0xc0039a0d80, 0xc0001f4a80)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:219 +0x15d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelineCallback.func1(0xc0039a0cc0)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:354 +0x7d
algod(11820) : github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch.func2(0xc003659d70, 0xc0039a0cc0, 0xc0000fa620)
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:398 +0x83
algod(11820) : created by github.com/[secure]/go-[secure]/catchup.(*Service).pipelinedFetch
algod(11820) : github.com/[secure]/go-[secure]/catchup/service.go:395 +0x238
algod(11820) : goroutine 13132 [select]:
algod(11820) : net/http.(*persistConn).writeLoop(0xc000c06d80)
algod(11820) : net/http/transport.go:2277 +0x1c3
algod(11820) : created by net/http.(*Transport).dialConn
algod(11820) : net/http/transport.go:1648 +0xc34
algod(11820) : goroutine 16079 [select]:
algod(11820) : database/sql.(*DB).connectionOpener(0xc003016180, 0x20bf780, 0xc000ef6d00)
algod(11820) : database/sql/sql.go:1071 +0x140
algod(11820) : created by database/sql.OpenDB
algod(11820) : database/sql/sql.go:742 +0x25d
algod(11820) : goroutine 13131 [IO wait]:
algod(11820) : internal/poll.runtime_pollWait(0x7fd68e080848, 0x72, 0x20a4b20)
algod(11820) : runtime/netpoll.go:203 +0x55
algod(11820) : internal/poll.(*pollDesc).wait(0xc0048c3618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
algod(11820) : internal/poll/fd_poll_runtime.go:87 +0xe4
algod(11820) : internal/poll.(*pollDesc).waitRead(...)
algod(11820) : internal/poll/fd_poll_runtime.go:92
algod(11820) : internal/poll.(*FD).Read(0xc0048c3600, 0xc0039ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
algod(11820) : internal/poll/fd_unix.go:169 +0x253
algod(11820) : net.(*netFD).Read(0xc0048c3600, 0xc0039ab000, 0x1000, 0x1000, 0x1, 0x20b5340, 0x37538a0)
algod(11820) : net/fd_unix.go:202 +0x66
algod(11820) : net.(*conn).Read(0xc003d80438, 0xc0039ab000, 0x1000, 0x1000, 0x10, 0x10, 0xc000c0d968)
algod(11820) : net/net.go:184 +0xa1
algod(11820) : net/http.(*persistConn).Read(0xc000c06d80, 0xc0039ab000, 0x1000, 0x1000, 0x0, 0xf7b, 0x1)
algod(11820) : net/http/transport.go:1825 +0xe8
algod(11820) : bufio.(*Reader).fill(0xc0032726c0)
algod(11820) : bufio/bufio.go:100 +0x19a
algod(11820) : bufio.(*Reader).Peek(0xc0032726c0, 0x1, 0x2, 0x0, 0x0, 0xc00385ea00, 0x0)
algod(11820) : bufio/bufio.go:138 +0xa0
algod(11820) : net/http.(*persistConn).readLoop(0xc000c06d80)
algod(11820) : net/http/transport.go:1978 +0x2af
algod(11820) : created by net/http.(*Transport).dialConn
algod(11820) : net/http/transport.go:1647 +0xc0f
algod(11820) : goroutine 12356 [semacquire]:
algod(11820) : sync.runtime_SemacquireMutex(0xc00022568c, 0x900000000, 0x1)
algod(11820) : runtime/sema.go:71 +0x47
algod(11820) : sync.(*Mutex).lockSlow(0xc000225688)
algod(11820) : sync/mutex.go:138 +0x1c1
algod(11820) : sync.(*Mutex).Lock(0xc000225688)
algod(11820) : sync/mutex.go:81 +0x7d
algod(11820) : github.com/[secure]/go-deadlock.lock.func1(0xc002fcbe70, 0xc00476f860)
algod(11820) : github.com/[secure]/go-deadlock@v0.2.1/deadlock.go:205 +0x35
algod(11820) : created by github.com/[secure]/go-deadlock.lock
algod(11820) : github.com/[secure]/go-deadlock@v0.2.1/deadlock.go:204 +0x136
algod(11820) :
algod(11820) :
algod(11820) :
algod(11820) :
algod(11820) :
algod(11820) :
algod(11820) :
algod(11820) : panic: (*logrus.Entry) (0x1b4be00,0xc003944b60) [recovered]
algod(11820) : panic: (*logrus.Entry) (0x1b4be00,0xc003944b60)
algod(11820) : goroutine 16827 [running]:
algod(11820) : github.com/[secure]/go-[secure]/logging.logger.Panic.func1(0xc0001407e0, 0xc0000be558)
algod(11820) : github.com/[secure]/go-[secure]/logging/log.go:252 +0xd5
algod(11820) : panic(0x1b4be00, 0xc003944b60)
algod(11820) : runtime/panic.go:975 +0x3e3
algod(11820) : github.com/sirupsen/logrus.Entry.log(0xc000140770, 0xc0039606f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
algod(11820) : github.com/sirupsen/logrus@v1.4.2/entry.go:239 +0x56a
algod(11820) : github.com/sirupsen/logrus.(*Entry).Log(0xc003944af0, 0x0, 0xc0035e3d90, 0x1, 0x1)
algod(11820) : github.com/sirupsen/logrus@v1.4.2/entry.go:268 +0x15f
algod(11820) : github.com/sirupsen/logrus.(*Entry).Panic(0xc003944af0, 0xc0035e3d90, 0x1, 0x1)
algod(11820) : github.com/sirupsen/logrus@v1.4.2/entry.go:306 +0x63
algod(11820) : github.com/[secure]/go-[secure]/logging.logger.Panic(0xc0001407e0, 0xc0000be558, 0xc0035e3d90, 0x1, 0x1)
algod(11820) : github.com/[secure]/go-[secure]/logging/log.go:256 +0x1ca
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod.(*deadlockLogger).Panic(...)
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/deadlockLogger.go:41
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod.(*deadlockLogger).onPotentialDeadlock.func1.1(0xc00007bc00, 0xc003913000, 0xf20)
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/deadlockLogger.go:85 +0xf7
algod(11820) : created by github.com/[secure]/go-[secure]/daemon/algod.(*deadlockLogger).onPotentialDeadlock.func1
algod(11820) : github.com/[secure]/go-[secure]/daemon/algod/deadlockLogger.go:83 +0x204
```
## Acceptance criteria
Reproduce failure. If the failure still exist then a follow up ticket should be created.
Contributor guide
Assessment
This issue has not been assessed yet.