Table metrics publisher timing out
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### Description / Background
Experiencing execution timeouts when metrics produced for multiple tables with many file references per partition. Loading StateStoreFiles snapshot for each table was taking ~26 secs. Had 3 tables in the instance so this exceeded the timeout (60 secs).
Increased the memory limit of the lambda to fix manually. Doing this also increases the amount of CPU available, reducing the time taken to a few seconds so the publisher finishes within the timeout.
### Technical Notes / Implementation Details
The lambda is currently hard coded to 1024 MB. Add an instance property so that this can be overridden.
We could also add instance properties for the timeout and schedule period. We can review how we've done this for other lambdas that fan out per Sleeper table in the same way.
Contributor guide
Assessment
This issue has not been assessed yet.