apache / apache/kyuubi

[Bug] SparkTBinaryFrontendService: java.net.BindException: Address already in use (Bind failed)

Open
#7,193 1 comment 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug
This error occurs occasionally when use beeline-mode in spark engine:
```
ERROR SparkTBinaryFrontendService: java.net.BindException: Address already in use (Bind failed)
Diagnostic: Failed to initialize SparkSQLEngine: Failed to initialize frontend service on xxx/xx.xx.xx.xx:0.
org.apache.kyuubi.KyuubiException: Failed to initialize SparkSQLEngine: Failed to initialize frontend service on xxx/xx.xx.xx.xx:0.
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:357)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:342)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:342)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)
Caused by: org.apache.kyuubi.KyuubiException: Failed to initialize frontend service on xxx/xx.xx.xx.xx:0.
at org.apache.kyuubi.service.TBinaryFrontendService.initialize(TBinaryFrontendService.scala:122)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:48)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.initialize(SparkSQLEngine.scala:70)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:353)
... 10 more
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:513)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at java.net.ServerSocket.(ServerSocket.java:237)
at org.apache.kyuubi.service.TBinaryFrontendService.initialize(TBinaryFrontendService.scala:100)
... 19 more
```
### Affects Version(s)

1.9.2

### Kyuubi Server Log Output

```logtalk

```

### Kyuubi Engine Log Output

```logtalk

```

### Kyuubi Server Configurations

```yaml

```

### Kyuubi Engine Configurations

```yaml

```

### Additional context

_No response_

### Are you willing to submit PR?

- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start with SparkSQLEngine.scala and service/TBinaryFrontendService.scala at the stack-trace locations where the frontend is initialized and the ServerSocket is bound. Reproduce the intermittent beeline-mode failure using the reported Spark engine path and inspect the available engine and server logs; done means the engine can initialize without the reported address-in-use failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.