estatsd impact the chef server performance and estatsd can not be disabled by estatsd['enable'] = false
Open
Nobody has claimed this yet.
Component: opscode-erchef
Status: To be prioritized
Triage: Try Reproducing
Type: Bug
Type: Enhancement
- Dominant language
- Erlang
- Stars
- 303
- Forks
- 211
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 5
Description
I encounter 2 issues, anyone have any idea?
- estatsd can not be disabled by estatsd['enable'] = false
- estatsd impact the chef server performance badly when setting to emit metric to server other than localhost, got error like this:
2018-05-22 13:48:37 =CRASH REPORT====
crasher:
initial call: stats_hero:init/1
pid: <0.1748.0>
registered_name: []
exception exit: {{timeout,{gen_server,call,[<0.1030.0>,{send,[[[["chefAPI",".application.allRequests"],58,"1",124,"c","\n"],[["chefAPI",".","",".allRequests"],58,"1",124,"c","\n"],[["chefAPI",".application.byRequestType.","clients",".","GET"],58,"1",124,"c","\n"]]]}]}},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,352}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
ancestors: [stats_hero_worker_sup,stats_hero_sup,<0.1020.0>]
messages: []
links: [<0.1044.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 987
stack_size: 27
reductions: 2626
neighbours:
2018-05-22 13:48:37 =SUPERVISOR REPORT====
Supervisor: {local,stats_hero_worker_sup}
Context: child_terminated
Reason: {timeout,{gen_server,call,[<0.1030.0>,{send,[[[["chefAPI",".application.allRequests"],58,"1",124,"c","\n"],[["chefAPI",".","",".allRequests"],58,"1",124,"c","\n"],[["chefAPI",".application.byRequestType.","clients",".","GET"],58,"1",124,"c","\n"]]]}]}}
Offender: [{pid,<0.1748.0>},{id,stats_hero},{mfargs,{stats_hero,start_link,undefined}},{restart_type,temporary},{shutdown,brutal_kill},{child_type,worker}]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the estatsd configuration, especially estatsd['enable'], and the stats_hero worker shown in the crash report. Reproduce the disabled setting and the timeout when emitting metrics to a non-localhost server. Done means estatsd can be disabled and metric delivery failures no longer badly affect Chef Server performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100