[Bug] SparkTBinaryFrontendService: java.net.BindException: Address already in use (Bind failed)
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
This error occurs occasionally when use beeline-mode in spark engine:
```
ERROR SparkTBinaryFrontendService: java.net.BindException: Address already in use (Bind failed)
Diagnostic: Failed to initialize SparkSQLEngine: Failed to initialize frontend service on xxx/xx.xx.xx.xx:0.
org.apache.kyuubi.KyuubiException: Failed to initialize SparkSQLEngine: Failed to initialize frontend service on xxx/xx.xx.xx.xx:0.
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:357)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:342)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:342)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)
Caused by: org.apache.kyuubi.KyuubiException: Failed to initialize frontend service on xxx/xx.xx.xx.xx:0.
at org.apache.kyuubi.service.TBinaryFrontendService.initialize(TBinaryFrontendService.scala:122)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:48)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.initialize(SparkSQLEngine.scala:70)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:353)
... 10 more
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:513)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at java.net.ServerSocket.(ServerSocket.java:237)
at org.apache.kyuubi.service.TBinaryFrontendService.initialize(TBinaryFrontendService.scala:100)
... 19 more
```
### Affects Version(s)
1.9.2
### Kyuubi Server Log Output
```logtalk
```
### Kyuubi Engine Log Output
```logtalk
```
### Kyuubi Server Configurations
```yaml
```
### Kyuubi Engine Configurations
```yaml
```
### Additional context
_No response_
### Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
Contributor guide
Research direction
Start with SparkSQLEngine.scala and service/TBinaryFrontendService.scala at the stack-trace locations where the frontend is initialized and the ServerSocket is bound. Reproduce the intermittent beeline-mode failure using the reported Spark engine path and inspect the available engine and server logs; done means the engine can initialize without the reported address-in-use failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100