awslabs / awslabs/amazon-kinesis-client-python

Checkpoint fails during handling signal (SIGINT)

Open
#117 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
377
Forks
228
PR merge metrics
No merged PRs in 30d

Description

I wanted to clear some records in buffer before the KCL (python) application terminates. So i was using signal to call my functions to clear my records and checkpoint again manually.
My observation is checkpoint fails when there is no data in shards and sometimes works when data is available in shards.

We are using Python KCL similar to the sample code below:
(https://github.com/awslabs/amazon-kinesis-client-python/blob/master/samples/sample_kclpy_app.py )

Sample Code to replicate the checkpoint issue

import signal

TERMINATE_SIGNALS = [
signal.SIGINT # 2
]

for sig in TERMINATE_SIGNALS:
signal.signal(sig, final_function)

def final_function():
clear_records()
force_checkpoint()
if args:
LOGGER.info("Shutting Down...")
sys.exit(args[0])

def clear_records():
print("Some Custom Function")

def force_checkpoint():
checkpoint(input_checkpointer, str(_largest_seq[0]), _largest_seq[1])

-- input_checkpointer = process_records_input.checkpointer

Error Logs 1:

2020-06-10 11:05:02,549 [multi-lang-daemon-0003] ERROR s.a.k.multilang.DrainChildSTDERRTask [NONE] - Received error line from subprocess [RuntimeError: reentrant call inside <_io.BufferedReader name=''>] for shard shardId-000000000000
Traceback (most recent call last):
File "consumer.py", line 148, in
kcl_process.run()
File "/home/ec2-user/.local/share/virtualenvs/test-1.0-Uc50fpnY/lib/python3.7/site-packages/amazon_kclpy/kcl.py", line 299, in run
line = self.io_handler.read_line()
File "/home/ec2-user/.local/share/virtualenvs/test-1.0-Uc50fpnY/lib/python3.7/site-packages/amazon_kclpy/kcl.py", line 68, in read_line
return self.input_file.readline()
...
File "/home/ec2-user/.local/share/virtualenvs/test-1.0-Uc50fpnY/lib/python3.7/site-packages/amazon_kclpy/kcl.py", line 133, in checkpoint
action = self._get_action()
File "/home/ec2-user/.local/share/virtualenvs/test-1.0-Uc50fpnY/lib/python3.7/site-packages/amazon_kclpy/kcl.py", line 117, in _get_action
line = self.io_handler.read_line()
File "/home/ec2-user/.local/share/virtualenvs/test-1.0-Uc50fpnY/lib/python3.7/site-packages/amazon_kclpy/kcl.py", line 68, in read_line
return self.input_file.readline()
RuntimeError: reentrant call inside <_io.BufferedReader name=''>

2.
INFO s.a.kinesis.multilang.LineReaderTask [NONE] - Stopping: Reading STDERR for shardId-000000000001
INFO s.a.k.m.MultiLangShardRecordProcessor [NONE] - Shutdown is requested.
INFO s.a.k.m.MultiLangShardRecordProcessor [NONE] - Requesting a checkpoint on shutdown notification.
INFO s.a.kinesis.multilang.MessageWriter [NONE] - Writing ShutdownRequestedMessage to child process for shard shardId-000000000001
INFO s.a.kinesis.multilang.LineReaderTask [NONE] - Starting: Reading next message from STDIN for shardId-000000000001
INFO s.a.kinesis.multilang.MessageWriter [NONE] - Writing CheckpointMessage to child process for shard shardId-000000000001
INFO s.a.kinesis.multilang.MessageWriter [NONE] - Cannot write message {"action":"checkpoint","sequenceNumber":"49604799652387804425055318128902113966719072308064944146","subSequenceNumber":616,"error":null} because writer is closed for shard shardId-000000000001
INFO s.a.kinesis.multilang.MessageWriter [NONE] - Writing CheckpointMessage to child process for shard shardId-000000000001
INFO s.a.kinesis.multilang.MessageWriter [NONE] - Cannot write message {"action":"checkpoint","sequenceNumber":"49604799652387804425055318128902113966719072308064944146","subSequenceNumber":616,"error":"IllegalStateException"} because writer is closed for shard shardId-000000000001

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.