Add client retry or throttling/backoff

Open
#39 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
ruby
Domain
api, cli, networking

Research direction

Start with the bundle exec neocities push ./_site entry point and reproduce the failure using the CircleCI configuration and a site containing about 100 small files. Confirm the behavior around the reported connection reset after rapid uploads; done means CI deployments complete reliably when uploads trigger throttling or transient disconnects.

Written by the indexing model from the issue text.

Description

Hey there!

Thank you for creating this tool! I've noticed when I use a hosted CI service like CircleCI that uploads a Jekyll site that sometimes it fails, and the client gets disconnected. When it does this, it looks like the push is screaming through (like >30 uploads in a second) and the client might be getting cut off and rate limited.

Example deploy command:

bundle exec neocities push ../_site

Error coming out, and retries seem to fail the same way after about 30 uploads that went through really fast:

Uploading tech/year/month/day/blah.html ... bundler: failed to load command: neocities (/home/circleci/project/neocities/vendor/bundle/ruby/3.1.0/bin/neocities)
/home/circleci/project/neocities/vendor/bundle/ruby/3.1.0/gems/httpclient-fixcerts-2.8.5/lib/httpclient/session.rb:813:in `rescue in block in parse_header': HTTPClient::KeepAliveDisconnected: Connection reset by peer (HTTPClient::KeepAliveDisconnected)

I haven't been able to replicate this issue from my laptop, where uploads of individual files are uploaded slower. If it is server-side rate limiting, it'd be nice if the client could jive with it. This doesn't fail every time with CI services. If things are a bit slower uploading from a CI service, it will sometimes work.

Steps to replicate
  • Have a repo with ~100 small files that are part of a static website (ie, Jekyll, Hugo).
  • Add a Gemfile that looks like this in the root directory of the repo:
source 'https://rubygems.org'

gem 'neocities'
  • Setup the repo to work with CircleCI
  • Add an environment variable to CircleCI project config called NEOCITIES_API_KEY with your API key
  • Add a CircleCI config to the repo .circleci/config.yml that looks something like this (assumes static assets are in a directory called _site like with Jekyll):
---
version: 2.1
orbs:
  ruby: circleci/ruby@2.0.0
ruby_image: &ruby_image cimg/ruby:3.1

workflows:
  build_and_deploy:
    jobs:
      - deploy_neocities

jobs:
  deploy_neocities:
    resource_class: large
    docker:
      - image: *ruby_image
    steps:
      - checkout
      - ruby/install-deps
      - run:
          name: Deploy Jekyll from _site directory to neocities
          command: |
            bundle exec neocities push ./_site       

With failed tasks, you can retry with SSH to log in with CircleCI to manually run the deploy command and experiment.

Dominant language
Ruby
Stars
70
Forks
20
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from neocities/neocities-ruby

All issues in neocities/neocities-ruby

Similar issues

More Ruby issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.