awslabs / awslabs/amazon-kinesis-client-nodejs

KCL nodejs with dynamodb local

Open
#400 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
304
Forks
198
Avg merge
36m
Merged PRs (30d)
2

Description

### Configuration

KCL nodejs version: `2.3.1`

- Env variables used:
```
AWS_ACCESS_KEY_ID=fake
AWS_SECRET_ACCESS_KEY=fake
```
- `.properties` configuration
```
executableName=/dist/kclClient/handler.js
streamName=arn:aws:dynamodb:ddblocal:000000000000:table/local_Dynamo-table/stream/2025-04-28T22:52:06.529
applicationName=KCL_local_Dynamo-table
initialPositionInStream=LATEST
dynamodbEndpoint=http://localhost:6007 // dynamodblocal instance runs on 6007
AwsCredentialsProvider=DefaultCredentialsProvider
```

- command to run
```
-e -p --java /usr/bin/java
```

### Issue

- The node-js client doesn't seem to working when using StreamARN that is created on DynamoDBlocal.
- Noticing that the error logs point to security token be not being valid.

```
2025-04-29 17:16:55,655 [multi-lang-daemon-0000] ERROR s.a.kinesis.coordinator.Scheduler [NONE] - Caught exception when initializing LeaseCoordinator
software.amazon.kinesis.leases.exceptions.DependencyException: software.amazon.awssdk.services.dynamodb.model.DynamoDbException: The security token included in the request is invalid. (Service: DynamoDb, Status Code: 400, Request ID: TCIHPG78RU1KGM97NVMUHSGM33VV4KQNSO5AEMVJF66Q9ASUAAJG)
at software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseRefresher.createTableIfNotExists(DynamoDBLeaseRefresher.java:355)
at software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseRefresher.createLeaseTableIfNotExists(DynamoDBLeaseRefresher.java:294)
at software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator.initialize(DynamoDBLeaseCoordinator.java:279)
at software.amazon.kinesis.coordinator.Scheduler.initialize(Scheduler.java:372)
at software.amazon.kinesis.coordinator.Scheduler.run(Scheduler.java:346)
at software.amazon.kinesis.multilang.MultiLangDaemon$MultiLangRunner.call(MultiLangDaemon.java:96)
at software.amazon.kinesis.multilang.MultiLangDaemon$MultiLangRunner.call(MultiLangDaemon.java:87)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```

### Followup

- I did check this issue mentioned on KCL repo
- [[Issue]( https://github.com/awslabs/amazon-kinesis-client/issues/652)]: However, this is very specific to `localstack` and not specific to `dynamodblocal`
- I did also checkout the following article about DynamoDB Streams Kinesis adapter [link](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.KCLAdapter.html). Does this mean, the kcl node.js library cannot be used for processing dynamo streams? If so, would appreciate if you could point the dynamo stream adapter for node.js(I know this is currently only available in java, are there any plans that we could expect in future)
- Could the issue thread be specific to node.js implementation 🙏.

Readers:
- 👍 if you've experienced something similar
- 👎 if you don't find this relevant — I'd appreciate hearing your thoughts either way

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the KCL bootstrap command with the shown .properties file, then inspect the failure at DynamoDBLeaseRefresher.createTableIfNotExists and the Node.js handler at /dist/kclClient/handler.js. Determine whether DynamoDB Local stream ARNs are supported by this client configuration and document the required setup or confirmed limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, javascript, nodejs
Domain
backend, databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.