awslabs / awslabs/amazon-kinesis-client-python
Recommended way of debugging a custom RecordProcessor
- Dominant language
- Python
- Stars
- 377
- Forks
- 228
- PR merge metrics
- No merged PRs in 30d
Description
Looking at `RecordProcessor::process_records`
```python
def process_records(self, process_records_input):
...
try:
for record in process_records_input.records:
...
except Exception as e:
sys.stderr.write("Encountered an exception while processing records. Exception was {e}\n".format(e=e))
```
I see that any exception raised in the `try` block will be written to stderr. However, the stderr buffer is not flushed immediately. In consequence, error messages are not displayed in the MultiLangDaemon's stdout, making debugging harder.
In your opinion, what would be the best solution to this problem? Here are some ideas:
- [disable output buffering](https://docs.python.org/2/using/cmdline.html#cmdoption-u) (`executableName = python -u myscript.py`)
- modify `RecordProcessor` so that it uses `amazon_kclpy.kcl._IOHandler`, e.g.:
```python
def process_records(self, process_records_input):
...
try:
for record in process_records_input.records:
...
except Exception as e:
# self.iohandler would have to be injected somewhere
self.iohandler.write_error("Encountered an exception while processing records. Exception was {e}\n".format(e=e))
```
- handle exceptions myself from `MyCustomRecordProcessor` at the `process_record` level:
```python
logger = logging.getLogger()
class MyCustomRecordProcessor(RecordProcessor):
def process_record(self, data, partition_key, sequence_number, sub_sequence_number):
try:
# some exception is raised here
except as e:
logger.error("Something wrong happened: {e}".format(e=e))
```
Contributor guide
Assessment
This issue has not been assessed yet.