wicketd requests to MGS fail with HTTP 400 and a 0-length response body
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- rust
- Domain
- networking
Research direction
Reproduce the next simultaneous dogfood-rack mupdate while capturing localhost traffic with snoop -o mgs-traffic.snoop -x0 -d lo0 port 12225. Compare the capture with the HTTP 400 failures and the absence of MGS log entries, then determine whether affected wicketd requests need retries. No source files or tests are named in the issue.
Written by the indexing model from the issue text.
Description
Today when trying to update 6 gimlets in the dogfood rack simultaneously, three of the updates failed partway through because wicketd got an HTTP 400 with no body response from MGS. There was no corresponding entry in the MGS logs, which makes it very likely hyper itself was sending the 400, which it does if it receives a non-HTTP request. https://github.com/hyperium/hyper/issues/3225 is a request for better after-the-fact debugging support from hyper, but in the meantime, the next time we try to mupdate the dogfood rack we should snoop the localhost traffic between wicketd and MGS in hopes of catching details on what's causing these 400s:
snoop -o mgs-traffic.snoop -x0 -d lo0 port 12225
We should (probably?) also add some kind of retries around at least some of the requests wicketd makes of MGS.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from oxidecomputer/omicron
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
oxidecomputer/omicron#11269 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
oxidecomputer/omicron#11266 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
oxidecomputer/omicron#11260 · 1 comment ·
-
wicket's errors should be better when trying to read sensitive data from ssh without a pseudo-tty Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
oxidecomputer/omicron#11148 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
oxidecomputer/omicron#10907 ·
All issues in oxidecomputer/omicron
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
kwakseongjae/auto-hwp#319 ·
-
area:cli bug filter-quality good first issue priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
bevyengine/bevy#25861 ·
-
comp-datalake
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
ClickHouse/ClickHouse#121222 ·
-
enhancement remote
Difficulty 2/5 1-3 hours Newbie friendliness 68/100