influxdata / influxdata/influxdb

Provide more directive error message for errors from queryd-adjacent components

Open
#19,031 1 comment 0 reactions 0 assignees View on GitHub
team/query
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

In a recent incident that was summarized here: https://github.com/influxdata/idpe/issues/7853,
we found unreliable behavior with query pods communicating to storage after storage pods are redeployed.

Specifically, we saw a lot of rpc errors:
![image.png](https://images.zenhubusercontent.com/5d63fb0b02ddc60b97fd9b37/486f917b-778b-4c8f-ae88-0e671af07a86)

We were spending time tracking down where did those error messages come from, and later we determined that it was actually from istio.

One of the take-aways from the investigation of this incident is that it will be very helpful if we could add additional information to those error messages stating which service we were trying to contact when we hit the error.

Contributor guide

Open the contributing guide

Research direction

Start by locating the queryd-adjacent components that produce RPC errors when query pods contact storage pods, then trace how the contacted service is known at the error site. Reproduce or inspect the redeployment communication failure described in the incident and confirm that the resulting error identifies the service being contacted.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, networking
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.