rubyforgood / rubyforgood/awbw

Orphaned Active Storage blobs: nothing reclaims unattached direct uploads

Open
#2,236 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
15
Forks
26
Avg merge
12h 42m
Merged PRs (30d)
242

Description

🤖 From Claude: Written up while reviewing #2217 (follow-ups to the file-upload answer type, #2136). Deliberately kept out of that PR — it's destructive and wants its own rollout.

The problem

Direct upload writes the blob to storage before the form is submitted. Anything that happens after that leaves a blob nothing references:

  • the registrant abandons the form
  • the content type is rejected
  • the file is over the 25 MB cap added in #2217
  • the registration fails for an unrelated reason (a too-long answer, a duplicate, a Stripe error)
  • a validation error elsewhere on the form and the registrant re-picks a different file

Nothing in the app reclaims these. There is no unattached, no purge_later, and no sweeper anywhere in app/, lib/, or config/.

StoryImportsController is a second, deliberate source: it stages a CSV as an unattached blob between preview and confirm, purging it explicitly on confirm or on a parse error. A preview that's never confirmed leaks one.

Why now

The exposure predates the file-upload feature — app/views/shared/_form_image_field.html.erb and app/views/assets/_primary_image_picker.html.erb both use direct_upload: true. But those are admin-only pages with a handful of uses a week. The public event registration form changes the volume by orders of magnitude, and production storage is DigitalOcean Spaces, where it's a line item.

Proposed fix

Solid Queue and config/recurring.yml are already in place with two daily jobs, so this is one job class and one YAML entry.

# app/jobs/purge_unattached_blobs_job.rb
class PurgeUnattachedBlobsJob < ApplicationJob
  queue_as :default

  # Direct upload stores a blob before its form is submitted, so a blob is
  # legitimately unattached while someone is still filling in the form (and
  # while a story CSV import sits between preview and confirm). Only sweep
  # ones old enough that no in-flight submission could still claim them.
  UNCLAIMED_AFTER = 2.days

  def perform
    ActiveStorage::Blob.unattached
      .where(active_storage_blobs: { created_at: ..UNCLAIMED_AFTER.ago })
      .find_each(&:purge_later)
  end
end
# config/recurring.yml, under production:
  purge_unattached_blobs:
    class: PurgeUnattachedBlobsJob
    schedule: every day at 4am

4am because 3:00 and 3:30 are already taken by reconcile_checkout_payments and issue_membership_invoices.

Details that matter

  • The age window is load-bearing, not a nicety. ActiveStorage::Blob.unattached is where.missing(:attachments), and a blob is supposed to be unattached during the gap between direct upload and submit. Sweeping without an age filter would delete files out from under registrants mid-form. Two days is Rails' own documented figure and comfortably exceeds any workflow here, including the story-import preview.
  • Qualify created_at with the table name. unattached builds a LEFT JOIN against active_storage_attachments, which also has a created_at — unqualified is ambiguous.
  • purge_later, not purge. Each deletion becomes its own retryable job, so one flaky Spaces call doesn't abort the sweep.
  • This is global and destructive. It reclaims any unattached blob, not just registration uploads. Re-check for deferred-attachment paths before it ships; the only one today is the story import described above, where sweeping is a fix rather than a conflict.

Suggested rollout

Ship it in report mode first — log the count and total bytes it would purge, run for a week, then look at the numbers before it deletes anything. "Delete N files" is not a change to land on an unexamined production dataset, and the first run is the one most likely to be surprising.

What this doesn't solve

The client-side size guard in #2217 reduces orphans (an oversized file never uploads), and accept helps for content type. Neither is enforceable — a crafted request can POST straight to /rails/active_storage/direct_uploads, which is unauthenticated by Rails default. The sweeper is the backstop. If abuse ever becomes real, the next lever is rate-limiting or authenticating that endpoint, which is a bigger change than it looks because the admin pages depend on it too.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing ActiveStorage::Blob.unattached, StoryImportsController, and the existing scheduled entries in config/recurring.yml. Add app/jobs/purge_unattached_blobs_job.rb and its production schedule, beginning in report mode as requested. Done means the age-qualified global sweep is observable, uses retryable purges, and has been checked against deferred-attachment paths before deletion is enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
rails, ruby, yaml
Domain
backend, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.