github / github/gh-ost

[bug] panic: runtime error: invalid memory address or nil pointer dereference

Open
#400 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

On a long running alter using the below setup after 20+ hours it get's an error with too many open files and if i try to query gh-ost via it's socket it will crash. Is this a known issue? I'm using the pre compiled version 1.0.36.

`./gh-ost --max-load=Threads_running=50,Threads_connected=700 --critical-load=Threads_running=3000,Threads_connected=5000 --chunk-size=1000 --throttle-control-replicas="10.20.2.4,10.30.2.12:3307,10.30.2.13,10.30.2.2,10.30.2.7,127.0.0.1:9901,127.0.0.1:9904,127.0.0.1:9905" --max-lag-millis=20000 --conf=admin.my --host=10.30.2.10 --database="database" --table="guest" --verbose --alter="engine=innodb" --cut-over=default --exact-rowcount --concurrent-rowcount --default-retries=240 --panic-flag-file=/tmp/gh-ost.panic.flag --postpone-cut-over-flag-file=/tmp/ghost.postpone.flag --execute`

```
Copy: 258060000/496528082 52.0%; Applied: 577507; Backlog: 100/100; Time: 21h32m30s(total), 21h32m27s(copy); streamer: mysql-bin.007696:756485258; State: throttled, 127.0.0.1:9905 replica-lag=924.999579s; ETA: 19h54m19s
2017-04-10 02:51:50 ERROR dial tcp 10.30.2.10:3306: socket: too many open files
2017-04-10 02:51:50 ERROR dial tcp 10.30.2.10:3306: socket: too many open files
2017-04-10 02:51:50 ERROR dial tcp 10.30.2.10:3306: socket: too many open files
2017-04-10 02:51:50 ERROR dial tcp 10.30.2.10:3306: socket: too many open files
2017-04-10 02:51:50 ERROR dial tcp 10.30.2.10:3306: socket: too many open files
2017-04-10 02:51:50 ERROR dial tcp 10.30.2.10:3306: socket: too many open files
2017-04-10 02:51:50 ERROR dial tcp 10.30.2.10:3306: socket: too many open files
2017-04-10 02:51:50 ERROR dial tcp 10.30.2.10:3306: socket: too many open files
2017-04-10 02:51:50 ERROR dial tcp 10.30.2.10:3306: socket: too many open files
2017-04-10 02:51:50 ERROR dial tcp 10.30.2.10:3306: socket: too many open files
2017-04-10 02:51:50 ERROR dial tcp 10.30.2.10:3306: socket: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
2017-04-10 02:51:50 ERROR accept unix /tmp/gh-ost.guest.sock: accept4: too many open files
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4a3f2d]

goroutine 1451637 [running]:
panic(0x6f6ac0, 0xc42000e0f0)
/usr/local/Cellar/go/1.7.4_2/libexec/src/runtime/panic.go:500 +0x1a1
github.com/github/gh-ost/go/logic.(*Server).handleConnection(0xc42020e740, 0x0, 0x0, 0x0, 0x0)
/Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/go/logic/server.go:101 +0x7d
created by github.com/github/gh-ost/go/logic.(*Server).Serve.func1
/Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/go/logic/server.go:81 +0x87
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4a3f2d]

goroutine 1451639 [running]:
panic(0x6f6ac0, 0xc42000e0f0)
/usr/local/Cellar/go/1.7.4_2/libexec/src/runtime/panic.go:500 +0x1a1
github.com/github/gh-ost/go/logic.(*Server).handleConnection(0xc42020e740, 0x0, 0x0, 0x0, 0x0)
/Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/go/logic/server.go:101 +0x7d
created by github.com/github/gh-ost/go/logic.(*Server).Serve.func1
/Users/shlomi-noach/dev/go/src/github.com/github/gh-ost/go/logic/server.go:81 +0x87
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.