grpc-ecosystem / grpc-ecosystem/grpc-spring
Race condition when finding an available port in GrpcServerProperties
- Dominant language
- Java
- Stars
- 3.7k
- Forks
- 858
- PR merge metrics
- No merged PRs in 30d
Description
**The context**
In our CI build we use port 0 for the grpc server, allowing the grpc server to determine an available port automatically.
However, we sometimes encounter an issue where the port is already bound when the grpc server starts.
**The bug**
This appears to be a result of the port assignment being non atomic:
1. `GrpcServerProperties::getPort` finds an available port (since the port is 0).
2. The socket used to find that port is closed.
3. The port is bound by some other thread / process on the machine.
4. When the grpc server starts and attempts to bind to the port, it fails since the port is already bound.
This is exacerbated by the `GrpcServerMetricAutoConfiguration::grpcInfoContributor` (which calls `GrpcServerProperties::getPort`) being loaded long before the call to
`GrpcServerLifecycle::start`, hence there is a long window during which the port can be bound by another thread / process on the machine.
**Stacktrace and logs**
```
[09:37:35][Step 2/3] Caused by: java.lang.IllegalStateException: Failed to start the grpc server
[09:37:35][Step 2/3] at net.devh.boot.grpc.server.serverfactory.GrpcServerLifecycle.start(GrpcServerLifecycle.java:74)
[09:37:35][Step 2/3] at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
[09:37:35][Step 2/3] ... 76 more
[09:37:35][Step 2/3] Caused by: java.io.IOException: Failed to bind to address 0.0.0.0/0.0.0.0:38840
[09:37:35][Step 2/3] at io.grpc.netty.NettyServer.start(NettyServer.java:328)
[09:37:35][Step 2/3] at io.grpc.internal.ServerImpl.start(ServerImpl.java:184)
[09:37:35][Step 2/3] at io.grpc.internal.ServerImpl.start(ServerImpl.java:93)
[09:37:35][Step 2/3] at net.devh.boot.grpc.server.serverfactory.GrpcServerLifecycle.createAndStartGrpcServer(GrpcServerLifecycle.java:113)
[09:37:35][Step 2/3] at net.devh.boot.grpc.server.serverfactory.GrpcServerLifecycle.start(GrpcServerLifecycle.java:72)
[09:37:35][Step 2/3] ... 77 more
[09:37:35][Step 2/3] Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
```
**The application's environment**
* Spring (boot): 2.7.9
* grpc-spring-boot-starter: 2.13.1.RELEASE
Contributor guide
Research direction
Start by reading GrpcServerProperties::getPort, GrpcServerMetricAutoConfiguration::grpcInfoContributor, and GrpcServerLifecycle::start/createAndStartGrpcServer. Trace how port 0 is selected and later bound, then investigate the reported CI failure and the interaction between metric loading and lifecycle startup. Done means the server can start reliably without the reported address-in-use race.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring-boot
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100