kragniz / kragniz/python-etcd3

Thread exception when calling close on client after watcher is started

Open
#997 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
450
Forks
194
PR merge metrics
No merged PRs in 30d

Description

**Description**:
The callback thread throws an exception when calling close on client if a watch has been started. Even if the watch is canceled the thread remains and throws a ValueError at etcd3/watch.py line 126.

**Code to reproduce**:
```
import etcd3

client = etcd3.client()
watch_id = client.watch_prefix('/')
client.cancel_watch(watch_id)

client.close()
```

**Error message**:
```
Exception in thread etcd3_watch_7f99646ef9b0:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/dist-packages/etcd3/watch.py", line 126, in _run
metadata=self._metadata)
File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 1084, in __call__
event_handler, self._context)
File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 1171, in create
operationses_and_tags, context)
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 478, in grpc._cython.cygrpc.Channel.integrated_call
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 296, in grpc._cython.cygrpc._integrated_call
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 222, in grpc._cython.cygrpc._call
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 262, in grpc._cython.cygrpc._call
ValueError: Cannot invoke RPC: Channel closed!
```

**python version**:
Python 3.6.9

**pip freeze**:
asn1crypto==0.24.0
coverage==5.0.3
crc32c==2.0
cryptography==2.1.4
etcd3==0.11.1
grpcio==1.26.0
idna==2.6
keyring==10.6.0
keyrings.alt==3.0
numpy==1.18.1
protobuf==3.11.2
pycrypto==2.6.1
pygobject==3.26.1
pyxdg==0.25
PyYAML==5.3
rdbd==0.0.1
SecretStorage==2.3.1
six==1.13.0
tenacity==6.0.0

**etcd --version**:
etcd Version: 3.2.17
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.10
Go OS/Arch: linux/amd64

**operating system**:
Docker version 19.03.5, build 633a0ea

**Dockerfile**
FROM ubuntu:18.04

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install --assume-yes \
etcd \
python3 \
python3-pip \
supervisor && \
mkdir /python

ENV DEBUG true
ENV ETCD_NAME provider
ENV ETCD_DATA_DIR /var/lib/etcd
ENV ETCD_INITIAL_CLUSTER_STATE new
ENV ETCD_INITIAL_CLUSTER_TOKEN etcd-cluster-01
ENV ETCDCTL_API=3
ENV CLUSTER_SEED_NODES node1,node2,node3

**etcd_start.sh**
```
#!/bin/bash

set -x
CURRENT_IP=$(hostname -i)
CLUSTER_SEED_NODES=${CLUSTER_SEED_NODES:-localhost}

INITIAL_CLUSTER=""
for NODE in $(echo ${CLUSTER_SEED_NODES} | sed 's/,/ /g'); do
NODE_ENTRY="${NODE}=http://${NODE}:2380"
if [[ -z "${INITIAL_CLUSTER}" ]]; then
INITIAL_CLUSTER="${NODE_ENTRY}"
else
INITIAL_CLUSTER="${INITIAL_CLUSTER},${NODE_ENTRY}"
fi
done

/usr/bin/etcd \
--name $(hostname) \
--initial-advertise-peer-urls "http://${CURRENT_IP}:2380" \
--listen-peer-urls "http://${CURRENT_IP}:2380" \
--listen-client-urls "http://${CURRENT_IP}:2379,http://127.0.0.1:2379" \
--advertise-client-urls "http://${CURRENT_IP}:2379" \
--initial-cluster-token etcd-cluster \
--initial-cluster ${INITIAL_CLUSTER}
```

Contributor guide

Open the contributing guide

Research direction

Start with etcd3/watch.py at _run around line 126, then trace the watch_prefix(), cancel_watch(), and client.close() paths using the provided reproduction. Check how the watcher thread is stopped before the gRPC channel closes. Done means running the example after canceling the watch and closing the client no longer produces the Channel closed exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.