Shopify / Shopify/shipit-engine

/events URL returns no content in 0.31.0

Open
#1,024 20 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
1.5k
Forks
154
Avg merge
2h 26m
Merged PRs (30d)
3

Description

We have just upgraded to 0.31.0 from 0.26.0 to pick up the recent GitHub API change.

The installation runs fine for several hours but then all requests will start timing out

It appears to be because an unbounded number of Pubsubstub connections are being created. I see many repeated events of the form

Feb 19 03:36:14 ip-172-31-37-230 env[1235]: I, [2020-02-19T03:36:14.855652 #1709]  INFO -- : [Pubsubstub::Subscription] Connecting client #7815470111917132360 (stack.1)

followed by an ever-growing number of repeated events of the form

Feb 19 03:39:19 ip-172-31-37-230 env[1235]: D, [2020-02-19T03:39:19.247287 #1728] DEBUG -- : [Pubsubstub::Subscription] Sending event #1582083559247 to client #7268750048476322816

This appears to be because the requests to /events?channels%5B%5D=stack.1 hang and return no content. In 0.26.0 the requests return content of the form:

id: 1582083655404
event: heartbeat
retry: 10000
data: ping

Reverting to 0.26.0 appears to fix the issue

Please let me know if I can provide any additional information to help diagnose the issue

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the /events?channels%5B%5D=stack.1 request and comparing its response with 0.26.0, then trace the Pubsubstub subscription behavior described in the logs. Done means /events returns heartbeat content and repeated requests no longer create an unbounded number of connections.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.