awslabs / awslabs/amazon-kinesis-client-python
Guidance on maxActiveThreads
- Dominant language
- Python
- Stars
- 377
- Forks
- 228
- PR merge metrics
- No merged PRs in 30d
Description
When we switched from using `maxActiveThreads = -1` to a small positive number (1, for example), kclpy stopped working. It would start up, but it would never create a python subprocess. If we made it 2, it would create a python subprocess, but never send it any messages. Etc.
It took me a while to come to a reasonable formula for maxActiveThreads, and I'm still not confident it's a good one. I figure
```
3 + (shardCount / workerCount)
```
where `workerCount` is the number of distinct workers (in our case, Kubernetes pods). `3` seems to be a magic overhead count, or threads that have to exist to manage processes on top of the actual python processes themselves.
I don't know if this formula is good or not, but either way I'd like to request appropriate guidance be added to documentation here, or in the sample properties file, so other engineers like me (for whom Java is not a first language) can make good decisions about how many threads they need.
Contributor guide
Assessment
This issue has not been assessed yet.