NYCPlanning / NYCPlanning/data-engineering
minio client downloads return 410; drop mc or repoint the install
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43
- Forks
- 3
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 44
Description
problem
dl.min.io returns 410 for every MinIO Client (mc) download path, and the response body says why:
The open-source MinIO Server, MinIO Client (mc) and MinIO KES projects are archived and no longer maintained. MinIO does not provide product support, security updates, or security advisories for them, and does not accept or process vulnerability reports concerning them.
So this isn't a stale pin or a moved URL. The project is archived and the distribution host is gone. Checked 2026-09-13:
| URL | status |
|---|---|
.../archive/mc.RELEASE.2020-04-19T19-17-53Z (what bash/utils.sh installs) |
410 |
.../archive/mc.RELEASE.2020-10-03T02-54-56Z (what ceqr installs) |
410 |
.../linux-amd64/mc (unpinned current) |
410 |
We have two install sites, and they fail differently:
bash/utils.sh:42usescurl -fL, so it fails loudly at download.products/ceqr/ceqr_app/bin/setup.sh:19usescurl -Owith no-f, so the 410 body (405 bytes of HTML) gets written to a file,chmod +x'd, and moved to/usr/bin/mc. curl exits 0. The failure surfaces later atmc config host addas a corrupt binary rather than as a download error.
This already broke the DevDB datasync workflow (run 34771273628). Fixed there by dropping the install, since that script's code paths never actually call mc: import_recipe only uses it on the private-ACL branch, and the datasets DevDB imports through it are public-read, so it takes the curl branch.
where mc is still used
| file | operations | live? |
|---|---|---|
bash/utils.sh:123,147 |
mc cat, mc cp in import_recipe, private-ACL branch only |
only caller was devdb_datasync.sh, which imports public datasets |
products/ceqr/ceqr_app/bin/config.sh |
mc stat, mc rm, mc cp, mc cat, mc ls throughout publish |
reached via ceqr publish, dispatch-only |
.github/workflows/on_hold/factfinder_acs.yml:73-74 |
mc cp |
on hold |
options
1. Repoint the install at GitHub releases. minio/mc still publishes there (latest RELEASE.2025-08-13T08-35-41Z), but assets are per-platform and per-release now (mc.linux-amd64.RELEASE.2025-08-13T08-35-41Z), so it needs a URL scheme change plus a five-year version bump. Smallest diff, but it keeps us on a dependency whose maintainers have explicitly said they no longer ship security updates for it.
2. Drop mc entirely. dcpy.utils.s3 already covers every operation we use:
| mc | dcpy.utils.s3 |
|---|---|
mc cp |
upload_file, download_file, copy_file |
mc ls |
get_filenames, list_objects, get_subfolders |
mc cat |
get_file_as_text, get_file_as_stream |
mc stat |
object_exists, get_metadata |
mc rm |
delete |
I'd go with 2. The credentials are already in the environment for boto, it consolidates on one S3 path instead of two, and it removes a dependency that just demonstrated it can disappear without warning.
worth checking before doing the work
- The
import_recipeprivate-ACL branch inbash/utils.shmay be dead rather than worth porting. Its only caller wasdevdb_datasync.sh, and everything that goes through it is public. - ceqr is where the actual porting work is. Before investing there, confirm
ceqr_publish.ymlstill runs: it does./ceqr setupwith noworking-directory, but the script lives atproducts/ceqr/ceqr_app/ceqr. ceqr_dep_monthly.ymlhas failed on the 1st of the month for at least July, August and September 2026, but on a different bug (TypeError: expected string or bytes-like object, got 'float'), not on mc. Separate issue.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with bash/utils.sh and products/ceqr/ceqr_app/bin/setup.sh and confirm whether ceqr_publish.yml still reaches the ceqr setup script. Trace the mc operations in ceqr_app/bin/config.sh against the listed dcpy.utils.s3 equivalents, and verify the public import path does not need mc. Done means the broken MinIO installs are removed or replaced and ceqr publishing still works without mc.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, python
- Domain
- cloud, devops
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100