awslabs / awslabs/amazon-kinesis-client-nodejs

Consuming multiple streams using KCL

Open
#4 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
304
Forks
198
Avg merge
36m
Merged PRs (30d)
2

Description

Hi I have an application that consuming multiple streams by starting several KCL processes, which works fine, but when the application is interrupted rather than an exception raised from my record processor those multiple instances of MultiLangDaemon will continue running, which causes problems.

When I restart the application a new set of MultiLangDaemon processes will be started and from this point on wards I am running into situation in which:
- either the DynamoDB table will contain a different worker ID i.e leaseOwner than my newly started worker's ID so nothing will happen,
- Or I can possibly alter the table and betting on MultiLangDaemon to repopulate the table with new leaseOwner ID but eventually the host machine will have many "java.exe" processes running and run out of memory.

I can stop those multiple node-js KCL bootstrap processes by executing OS commands in my application but I don't know how to ONLY stop MultiLangDaemon processes (they are just java.exe on windows).

All these can be resolved "relatively properly", if I can stop the MultiLangDaemon process, hence when new MultiLangDaemon process start the leaseOwner ID will be changed and the process starts normally and the machine won't run out of memory. (I can't just kill all "java.exe" on windows or java on Unix system)

You suggestion and help will be much appreciated !

Contributor guide

Open the contributing guide

Research direction

Start by examining how the application launches and handles interruption of multiple MultiLangDaemon processes. Reproduce an interrupted restart and determine how process ownership can distinguish these daemons from unrelated java.exe processes; done means they stop cleanly without leaving duplicate workers or exhausting memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, javascript
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.