awslabs / awslabs/amazon-kinesis-client-python

Guidance on maxActiveThreads

Open
#94 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
377
Forks
228
PR merge metrics
No merged PRs in 30d

Description

When we switched from using `maxActiveThreads = -1` to a small positive number (1, for example), kclpy stopped working. It would start up, but it would never create a python subprocess. If we made it 2, it would create a python subprocess, but never send it any messages. Etc.

It took me a while to come to a reasonable formula for maxActiveThreads, and I'm still not confident it's a good one. I figure

```
3 + (shardCount / workerCount)
```

where `workerCount` is the number of distinct workers (in our case, Kubernetes pods). `3` seems to be a magic overhead count, or threads that have to exist to manage processes on top of the actual python processes themselves.

I don't know if this formula is good or not, but either way I'd like to request appropriate guidance be added to documentation here, or in the sample properties file, so other engineers like me (for whom Java is not a first language) can make good decisions about how many threads they need.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.