cockroachdb / cockroachdb/cockroach
roachtest: datadog instrumentation across all TeamCity Builds & release branches
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
### Background
During an investigation (https://cockroachlabs.slack.com/archives/C03SG8QKYRJ/p1778270425439519), it was observed that expected test runner logs in datadog were not present. This issue is to investigate missing logs, and make the roachtest instrumentation consistent across branch versions and cloud provider CI, and to establish next steps.
### Goal
Roachtest operational logs (non test artifacts) are currently instrumented with Datadog for master (Test logs have been uploaded since either 25.4 or 26.1). As the capability for splitting roachtest operations and test execution logs already exists (https://github.com/cockroachdb/cockroach/pull/166175), all operational logs i.e. the test runner & cluster creation logs, should be uploaded to datadog.
* Note: Test logs across all branches and cloud provider CI is too expensive https://github.com/cockroachdb/cockroach/pull/163239#issuecomment-3930422144
Various backports need to be added across release branches, missing backports may be why the experience has been inconsistent. Also the Datadog API key needs to be across all TC Builds.
Then verify instrumentation is behaving as expected.
Related: https://github.com/cockroachdb/cockroach/issues/155854
Jira issue: CRDB-64157
Contributor guide
Assessment
This issue has not been assessed yet.