nacos-group / nacos-group/nacos-sdk-python

_server_request_watcher silently exits on RST_STREAM without triggering reconnection

Open
#294 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
471
Forks
155
PR merge metrics
No merged PRs in 30d

Description

Description

When the bidirectional gRPC stream encounters an RST_STREAM error (error code 8), the _server_request_watcher task exits silently without triggering any reconnection logic. This causes ephemeral instances to be removed from Nacos server after the heartbeat timeout, while the client remains unaware of the disconnection.

Environment
  • nacos-sdk-python version: 3.0.4
  • Python version: 3.13
  • Nacos server version: 3.1.1
  • Network: Behind AWS ALB Gateway
Steps to Reproduce
  1. Register an ephemeral instance using NacosNamingService
  2. Wait for network idle timeout (e.g., ALB 60s idle timeout) or simulate RST_STREAM
  3. Observe that the client's rpc_client_status remains RUNNING
  4. The instance disappears from Nacos registry after server-side heartbeat timeout
Expected Behavior

When the bidirectional stream dies, the SDK should:

  1. Detect the stream termination
  2. Update rpc_client_status to UNHEALTHY
  3. Trigger reconnection via reconnection_chan
  4. Re-register the ephemeral instance
Actual Behavior

The _server_request_watcher task in grpc_client.py exits silently:

async def _server_request_watcher(self, grpc_conn: GrpcConnection):
    async for payload in grpc_conn.bi_stream_send():
        try:
            # ... handle payload
        except Exception as e:
            self.logger.error(f"handle server request occur exception: {e}")

When RST_STREAM occurs, the async for loop simply terminates without:

  • Catching the stream termination
  • Updating client status
  • Triggering reconnection
Root Cause Analysis

The issue is in v2/nacos/transport/grpc_client.py:

  1. Line 93: Task is created via asyncio.create_task(self._server_request_watcher(grpc_conn))
  2. Lines 95-115: The async for loop has no finally block to handle stream termination
  3. The SDK's health check (send_health_check) uses unary requests, not the bidirectional stream, so it may still succeed even when the stream is dead
Proposed Fix
async def _server_request_watcher(self, grpc_conn: GrpcConnection):
    try:
        async for payload in grpc_conn.bi_stream_send():
            try:
                self.logger.info(
                    "receive stream server request, connection_id:%s, original info: %s"
                    % (grpc_conn.get_connection_id(), str(payload))
                )
                request = GrpcUtils.parse(payload)
                if request:
                    await self._handle_server_request(request, grpc_conn)
            except Exception as e:
                self.logger.error(
                    f"[{grpc_conn.connection_id}] handle server request occur exception: {e}"
                )
    except Exception as e:
        self.logger.error(
            f"[{grpc_conn.connection_id}] bidirectional stream error: {e}"
        )
    finally:
        # Trigger reconnection when stream ends unexpectedly
        if not self.is_shutdown() and not grpc_conn.is_abandon():
            self.logger.warning(
                f"[{grpc_conn.connection_id}] bidirectional stream ended, triggering reconnect"
            )
            self.rpc_client_status = RpcClientStatus.UNHEALTHY
            await self.reconnection_chan.put(
                ReconnectContext(server_info=None, on_request_fail=True)
            )
Workaround

We implemented a wrapper that monitors SDK internal state and forces reconnection:

def _check_sdk_internal_state(self) -> tuple[bool, str]:
    # Check rpc_client_status
    # Check current_connection exists
    # Check connection.abandon flag
    # Check if _server_request_task.done()  <-- Key detection
    ...

This works but requires accessing private attributes, which is fragile.

Impact

This issue affects all users who:

  • Use ephemeral instances (the default)
  • Operate behind load balancers or NAT with idle timeouts
  • Experience any network interruption that causes RST_STREAM

The service silently disappears from Nacos registry without any client-side error or recovery attempt.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read v2/nacos/transport/grpc_client.py, starting at _server_request_watcher and the task creation around line 93. Trace how stream termination, rpc_client_status, reconnection_chan, and grpc_conn.is_abandon() are handled. Reproduce an RST_STREAM or idle-timeout scenario and verify that unexpected termination marks the client UNHEALTHY, requests reconnection, and restores ephemeral registration.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, python
Domain
backend, networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.