couchbaselabs / couchbaselabs/perfcluster-aws

Make SG expvar collection more robust

Open
#42 0 comments 0 reactions 1 assignee Claimed by @sethrosetter View on GitHub
backlog bug
Dominant language
No language data
Stars
0
Forks
5
PR merge metrics
No merged PRs in 30d

Description

Right now if the SG machine is rebooted, the expvars will no longer be collected, even after restarting splunk via:

```
sudo /opt/splunkforwarder/bin/splunk start --accept-license
```

Instead of the current approach:

```
- name: Collect _expvar logs
shell: ((nohup /home/centos/collect_url_requests.sh -r http://localhost:4985/_expvar -f /home/centos/_expvar.logs -t 60 1>/dev/null 2>&1) &)
- name: Capture _expvar logs
shell: /opt/splunkforwarder/bin/splunk add monitor /home/centos/_expvar.logs -index main -sourcetype sync_gateway_expvars -auth {{ forward_server_auth }}
```

We should tell splunk to call a script every 60 seconds, which collects the expvars from http://localhost:4985/_expvar and sends the output to the splunk server. That way, as long as splunkd is running, we will be collecting expvars.

I guess it's possible to [modify the inputs.conf](https://github.com/couchbaselabs/perfcluster-aws/blob/master/ansible/playbooks/install-splunkforwarder.yml#L94-L96) file to add this entry, but that might be a bit tricky since we don't want to keep adding entries. Maybe a better approach would be to have an additional file like sg_inputs.conf with this directive and tell splunk to look at that as well.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.