linkedin / linkedin/Burrow

Burrow does not exit after panic

Open
#791 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

Version: 1.6.0
Issue: burrow hangs (stops responding, but does not exit) after a failure to unlock from zk:

```
2023-10-02 17:33:19.940 | {"level":"info","ts":1696257198.8336904,"msg":"re-submitting `0` credentials after reconnect","type":"coordinator","name":"zookeeper"}
2023-10-02 17:33:19.940 | {"level":"info","ts":1696257198.8336573,"msg":"authenticated: id=74567085257124526, timeout=6000","type":"coordinator","name":"zookeeper"}
2023-10-02 17:33:19.940 | {"level":"info","ts":1696257198.811102,"msg":"starting session","type":"coordinator","name":"zookeeper"}
2023-10-02 17:33:19.940 | {"level":"info","ts":1696257198.8110363,"msg":"Connected to [zk-ip1]:2181","type":"coordinator","name":"zookeeper"}
2023-10-02 17:33:18.938 | stderr /home/runner/work/Burrow/Burrow/core/internal/notifier/coordinator.go:272 +0x1f1
2023-10-02 17:33:18.938 | stderr created by github.com/linkedin/Burrow/core/internal/notifier.(*Coordinator).Start
2023-10-02 17:33:18.938 | stderr /home/runner/work/Burrow/Burrow/core/internal/notifier/coordinator.go:328 +0x505
2023-10-02 17:33:18.938 | stderr github.com/linkedin/Burrow/core/internal/notifier.(*Coordinator).manageEvalLoop(0xc0000f0380)
2023-10-02 17:33:18.934 | stderr goroutine 115 [running]:
2023-10-02 17:33:18.934 | stderr
2023-10-02 17:33:18.934 | stderr panic: Unable to release zookeeper lock after session expiration
```

Seems like that panic was somehow recovered, because [Burrow failed at](https://github.com/linkedin/Burrow/blob/v1.6.0/main.go#L52) was not printed. And the process did not died until 10 minutes later when I send it a SIGTERM.

A similar thing happens if I start it locally, without access to zk:

```
{"level":"panic","ts":1696264692.487353,"msg":"Failure to start zookeeper","type":"coordinator","name":"zookeeper","error":"lookup zk-host on [zk-ip]:53: no such host"}
panic: Failure to start zookeeper [recovered]
panic: Failure to start zookeeper

goroutine 1 [running]:
main.handleExit()
/home/runner/work/Burrow/Burrow/main.go:63 +0xf8
panic({0xbeb8a0, 0xc0003a60b0})
/opt/hostedtoolcache/go/1.20.1/x64/src/runtime/panic.go:884 +0x213
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x1?, 0x7f680a4c45e8?, {0x0?, 0x0?, 0xc000132020?})
/home/runner/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:198 +0x65
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00011c000, {0xc000226180, 0x1, 0x1})
/home/runner/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:264 +0x3ec
go.uber.org/zap.(*Logger).Panic(0xc000226000?, {0xd227e4?, 0x0?}, {0xc000226180, 0x1, 0x1})
/home/runner/go/pkg/mod/go.uber.org/zap@v1.24.0/logger.go:258 +0x59
github.com/linkedin/Burrow/core/internal/zookeeper.(*Coordinator).Start(0xc00014a240)
/home/runner/work/Burrow/Burrow/core/internal/zookeeper/coordinator.go:87 +0x42b
github.com/linkedin/Burrow/core.Start(0xc000084540?, 0xc0001a5ef0?)
/home/runner/work/Burrow/Burrow/core/burrow.go:158 +0x49b
main.main()
/home/runner/work/Burrow/Burrow/main.go:114 +0x4d2
```

And no logs since that, the prcess is alive. This time the panic clearly has been recovered.
I would very much like burrow to exit on network problems, so my orchestration could restart it.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing panic handling through main.go, core/burrow.go, core/internal/zookeeper/coordinator.go, and core/internal/notifier/coordinator.go, using the reported stack traces as entry points. Check how failures in the startup and unlock paths affect the process lifecycle. Done means Burrow exits on these network or lock failures so orchestration can restart it.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.