cockroachdb / cockroachdb/cockroach

release-26.3.2-rc: roachtest: dns_problem failed

Open
#174,804 1 comment 0 reactions 0 assignees View on GitHub
branch-release-26.3.2-rc O-roachtest O-robot T-testeng X-infra-flake
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.dns_problem [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21502674?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21502674?buildTab=artifacts#/cdc/rolling-restart) on release-26.3.2-rc @ [6149d98a3244ba2a3e4417d9e363da83e213e1b7](https://github.com/cockroachdb/cockroach/commits/6149d98a3244ba2a3e4417d9e363da83e213e1b7):

Failed with:

```
test cdc/rolling-restart failed: (test_monitor.go:35).func1: test monitor: EOF [owner=test-eng]
(cluster.go:2858).Start: TRANSIENT_ERROR(dns_problem): output: : signal: killed
test artifacts and logs in: /artifacts/cdc/rolling-restart/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21502674-1788757737-42-n6cpu4-0001 | | 10.91.99.170 |
| teamcity-21502674-1788757737-42-n6cpu4-0002 | | 10.91.100.230 |
| teamcity-21502674-1788757737-42-n6cpu4-0003 | | 10.91.99.241 |
| teamcity-21502674-1788757737-42-n6cpu4-0004 | | 10.91.99.216 |
| teamcity-21502674-1788757737-42-n6cpu4-0005 | | 10.91.99.93 |
| teamcity-21502674-1788757737-42-n6cpu4-0006 | | 10.91.97.146 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21502674/cdc-rolling-restart/1788766368124/1788766567943)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21502674-1788757737-42-n6cpu4&from_ts=1788766368124&to_ts=1788766567943&live=false&storage=flex_tier)

Same failure on other branches

- #174153 release-26.2: roachtest: dns_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-26.2]
- #137246 roachtest: dns_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]
- #125300 roachtest: dns_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]
- #121092 roachtest: dns_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]
- #121090 roachtest: dns_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]
- #120959 roachtest: dns_problem failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachlabs/test-eng

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*dns_problem.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68000

Contributor guide

Open the contributing guide

Research direction

Start with the linked TeamCity build log and artifacts for the failing cdc/rolling-restart roachtest, then consult the roachtest README. Determine why cluster startup reports dns_problem and whether the failure is reproducible; related failures are listed on other branches. Done means identifying and addressing the underlying cause, with the roachtest passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, go
Domain
infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.