couchbaselabs / couchbaselabs/perfcluster-aws
Make SG expvar collection more robust
- Dominant language
- No language data
- Stars
- 0
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
Right now if the SG machine is rebooted, the expvars will no longer be collected, even after restarting splunk via:
```
sudo /opt/splunkforwarder/bin/splunk start --accept-license
```
Instead of the current approach:
```
- name: Collect _expvar logs
shell: ((nohup /home/centos/collect_url_requests.sh -r http://localhost:4985/_expvar -f /home/centos/_expvar.logs -t 60 1>/dev/null 2>&1) &)
- name: Capture _expvar logs
shell: /opt/splunkforwarder/bin/splunk add monitor /home/centos/_expvar.logs -index main -sourcetype sync_gateway_expvars -auth {{ forward_server_auth }}
```
We should tell splunk to call a script every 60 seconds, which collects the expvars from http://localhost:4985/_expvar and sends the output to the splunk server. That way, as long as splunkd is running, we will be collecting expvars.
I guess it's possible to [modify the inputs.conf](https://github.com/couchbaselabs/perfcluster-aws/blob/master/ansible/playbooks/install-splunkforwarder.yml#L94-L96) file to add this entry, but that might be a bit tricky since we don't want to keep adding entries. Maybe a better approach would be to have an additional file like sg_inputs.conf with this directive and tell splunk to look at that as well.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.