NYCPlanning / NYCPlanning/data-engineering

minio client downloads return 410; drop mc or repoint the install

Open
#2,627 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

platform
Dominant language
Python
Stars
43
Forks
3
Avg merge
23h 3m
Merged PRs (30d)
44

Description

problem

dl.min.io returns 410 for every MinIO Client (mc) download path, and the response body says why:

The open-source MinIO Server, MinIO Client (mc) and MinIO KES projects are archived and no longer maintained. MinIO does not provide product support, security updates, or security advisories for them, and does not accept or process vulnerability reports concerning them.

So this isn't a stale pin or a moved URL. The project is archived and the distribution host is gone. Checked 2026-09-13:

URL status
.../archive/mc.RELEASE.2020-04-19T19-17-53Z (what bash/utils.sh installs) 410
.../archive/mc.RELEASE.2020-10-03T02-54-56Z (what ceqr installs) 410
.../linux-amd64/mc (unpinned current) 410

We have two install sites, and they fail differently:

  • bash/utils.sh:42 uses curl -fL, so it fails loudly at download.
  • products/ceqr/ceqr_app/bin/setup.sh:19 uses curl -O with no -f, so the 410 body (405 bytes of HTML) gets written to a file, chmod +x'd, and moved to /usr/bin/mc. curl exits 0. The failure surfaces later at mc config host add as a corrupt binary rather than as a download error.

This already broke the DevDB datasync workflow (run 34771273628). Fixed there by dropping the install, since that script's code paths never actually call mc: import_recipe only uses it on the private-ACL branch, and the datasets DevDB imports through it are public-read, so it takes the curl branch.

where mc is still used

file operations live?
bash/utils.sh:123,147 mc cat, mc cp in import_recipe, private-ACL branch only only caller was devdb_datasync.sh, which imports public datasets
products/ceqr/ceqr_app/bin/config.sh mc stat, mc rm, mc cp, mc cat, mc ls throughout publish reached via ceqr publish, dispatch-only
.github/workflows/on_hold/factfinder_acs.yml:73-74 mc cp on hold

options

1. Repoint the install at GitHub releases. minio/mc still publishes there (latest RELEASE.2025-08-13T08-35-41Z), but assets are per-platform and per-release now (mc.linux-amd64.RELEASE.2025-08-13T08-35-41Z), so it needs a URL scheme change plus a five-year version bump. Smallest diff, but it keeps us on a dependency whose maintainers have explicitly said they no longer ship security updates for it.

2. Drop mc entirely. dcpy.utils.s3 already covers every operation we use:

mc dcpy.utils.s3
mc cp upload_file, download_file, copy_file
mc ls get_filenames, list_objects, get_subfolders
mc cat get_file_as_text, get_file_as_stream
mc stat object_exists, get_metadata
mc rm delete

I'd go with 2. The credentials are already in the environment for boto, it consolidates on one S3 path instead of two, and it removes a dependency that just demonstrated it can disappear without warning.

worth checking before doing the work

  • The import_recipe private-ACL branch in bash/utils.sh may be dead rather than worth porting. Its only caller was devdb_datasync.sh, and everything that goes through it is public.
  • ceqr is where the actual porting work is. Before investing there, confirm ceqr_publish.yml still runs: it does ./ceqr setup with no working-directory, but the script lives at products/ceqr/ceqr_app/ceqr.
  • ceqr_dep_monthly.yml has failed on the 1st of the month for at least July, August and September 2026, but on a different bug (TypeError: expected string or bytes-like object, got 'float'), not on mc. Separate issue.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with bash/utils.sh and products/ceqr/ceqr_app/bin/setup.sh and confirm whether ceqr_publish.yml still reaches the ceqr setup script. Trace the mc operations in ceqr_app/bin/config.sh against the listed dcpy.utils.s3 equivalents, and verify the public import path does not need mc. Done means the broken MinIO installs are removed or replaced and ceqr publishing still works without mc.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, python
Domain
cloud, devops
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.