opensearch-project / opensearch-project/OpenSearch
[BUG] TaskResourceTrackingService consuming more CPU than expected
@ansjcy is already working on this.
Since Nov 13, 2024.
- Dominant language
- Java
- Stars
- 13.7k
- Forks
- 3k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 108
Description
Describe the bug
I profiled an OpenSearch server running the term query operation from the big5 OSB workload. This query is very fast, so the intent was to find any overhead not related to doing the actual work of searching indexes that could be optimized. The surprising finding is that TaskResourceTrackingService takes a little over 7% of the total CPU cycles. A big chunk of that work is simply marshaling the TaskResourceInfo object to and from a JSON string (in getTaskResourceUsageFromThreadContext and writeTaskResourceUsage()).
Related component
Search:Performance
To Reproduce
This overhead will happen on any search, though for more expensive searches it may be less noticeable as the CPU will be dominated by other work.
Expected behavior
Assuming that the TaskResourceInfo is just being serialized for machine-to-machine communication, then it should use an efficient binary serialization to avoid the XContent/Jackson/JSON overhead in the hot path on searches.
Additional Details
Here is a zoomed-in snippet of a profile showing getTaskResourceUsageFromThreadContext():
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.