awslabs / awslabs/amazon-kinesis-client-python
Checkpoint fails during handling signal (SIGINT)
- Dominant language
- Python
- Stars
- 377
- Forks
- 228
- PR merge metrics
- No merged PRs in 30d
Description
I wanted to clear some records in buffer before the KCL (python) application terminates. So i was using signal to call my functions to clear my records and checkpoint again manually.
My observation is checkpoint fails when there is no data in shards and sometimes works when data is available in shards.
We are using Python KCL similar to the sample code below:
(https://github.com/awslabs/amazon-kinesis-client-python/blob/master/samples/sample_kclpy_app.py )
Sample Code to replicate the checkpoint issue
import signal
TERMINATE_SIGNALS = [
signal.SIGINT # 2
]
for sig in TERMINATE_SIGNALS:
signal.signal(sig, final_function)
def final_function():
clear_records()
force_checkpoint()
if args:
LOGGER.info("Shutting Down...")
sys.exit(args[0])
def clear_records():
print("Some Custom Function")
def force_checkpoint():
checkpoint(input_checkpointer, str(_largest_seq[0]), _largest_seq[1])
-- input_checkpointer = process_records_input.checkpointer
Error Logs 1:
2020-06-10 11:05:02,549 [multi-lang-daemon-0003] ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess [RuntimeError: reentrant call inside <_io.BufferedReader name=''>] for shard shardId-000000000000
Traceback (most recent call last):
File "consumer.py", line 148, in
kcl_process.run()
File "/home/ec2-user/.local/share/virtualenvs/test-1.0-Uc50fpnY/lib/python3.7/site-packages/amazon_kclpy/kcl.py", line 299, in run
line = self.io_handler.read_line()
File "/home/ec2-user/.local/share/virtualenvs/test-1.0-Uc50fpnY/lib/python3.7/site-packages/amazon_kclpy/kcl.py", line 68, in read_line
return self.input_file.readline()
...
File "/home/ec2-user/.local/share/virtualenvs/test-1.0-Uc50fpnY/lib/python3.7/site-packages/amazon_kclpy/kcl.py", line 133, in checkpoint
action = self._get_action()
File "/home/ec2-user/.local/share/virtualenvs/test-1.0-Uc50fpnY/lib/python3.7/site-packages/amazon_kclpy/kcl.py", line 117, in _get_action
line = self.io_handler.read_line()
File "/home/ec2-user/.local/share/virtualenvs/test-1.0-Uc50fpnY/lib/python3.7/site-packages/amazon_kclpy/kcl.py", line 68, in read_line
return self.input_file.readline()
RuntimeError: reentrant call inside <_io.BufferedReader name=''>
2.
INFO s.a.kinesis.multilang.LineReaderTask [NONE] - Stopping: Reading STDERR for shardId-000000000001
INFO s.a.k.m.MultiLangShardRecordProcessor [NONE] - Shutdown is requested.
INFO s.a.k.m.MultiLangShardRecordProcessor [NONE] - Requesting a checkpoint on shutdown notification.
INFO s.a.kinesis.multilang.MessageWriter [NONE] - Writing ShutdownRequestedMessage to child process for shard shardId-000000000001
INFO s.a.kinesis.multilang.LineReaderTask [NONE] - Starting: Reading next message from STDIN for shardId-000000000001
INFO s.a.kinesis.multilang.MessageWriter [NONE] - Writing CheckpointMessage to child process for shard shardId-000000000001
INFO s.a.kinesis.multilang.MessageWriter [NONE] - Cannot write message {"action":"checkpoint","sequenceNumber":"49604799652387804425055318128902113966719072308064944146","subSequenceNumber":616,"error":null} because writer is closed for shard shardId-000000000001
INFO s.a.kinesis.multilang.MessageWriter [NONE] - Writing CheckpointMessage to child process for shard shardId-000000000001
INFO s.a.kinesis.multilang.MessageWriter [NONE] - Cannot write message {"action":"checkpoint","sequenceNumber":"49604799652387804425055318128902113966719072308064944146","subSequenceNumber":616,"error":"IllegalStateException"} because writer is closed for shard shardId-000000000001
Contributor guide
Assessment
This issue has not been assessed yet.