django / django/djangoproject.com

Add optional django-storages (S3) support for media

Open
#2,802 8 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2k
Forks
1.1k
Avg merge
2d 7h
Merged PRs (30d)
26

Description

Uploaded media and release artifacts currently live on the server's local file system (`MEDIA_ROOT = DATA_DIR / "media"` in `prod.py`), which must be copied when replacing the production server. This issue adds **optional** S3 storage via [django-storages](https://django-storages.readthedocs.io/), selected by environment configuration. Local file system storage remains the default; the feature is enabled from the deployment repo once this lands.

## Changes (this repo)

- **Dependency**: add `django-storages[boto3]` (prod group in `pyproject.toml`, regenerate `requirements/prod.txt`).
- **Settings** (`djangoproject/settings/prod.py`): select `STORAGES["default"]`'s backend based on environment configuration (e.g., use `storages.backends.s3.S3Storage` when the bucket name is set, otherwise keep `FileSystemStorage`). All S3 config (bucket, region) comes from the environment / `secrets.json`; nothing hardcoded. The EC2 instance will have a role with S3 access, so no explicit credentials are required.
- **Release artifacts** (`releases/models.py`): `get_storage()` explicitly returns `FileSystemStorage(allow_overwrite=True)`; make it return the S3 equivalent (with `allow_overwrite=True`) when S3 is enabled, so artifact uploads keep their exact names.
- **Docs release zips** (`docs/management/commands/build_doc_release.py`): the docs HTML zip is written directly to `settings.MEDIA_ROOT / "docs"`; write it through the default storage instead so it lands in S3 when enabled.
- **sorl-thumbnail** (fundraising logos, `djangoproject/thumbnails.py`): verify `get_thumbnail()` works with an S3 source/storage, and configure `THUMBNAIL_STORAGE` so generated thumbnails are written to the bucket.
- **Tests**: suite must pass without AWS credentials (default configuration unchanged); add coverage for the S3 configuration path (backend selection and file operations against a mocked S3 backend, e.g. `moto` or `django-storages`'s in-memory S3).

## Out of scope

- **Enabling in the deployment repo** (private Ansible repo): S3 bucket, the env vars that turn the feature on, one-time copy of existing `media/` contents, nginx changes for `media.djangoproject.com` (including the pre-1.4 `/css|img/` redirect), and dropping the `media/` directory from automated backups.
- **Trac**: config and attachments stay on the file system.
- **Docs rendering**: `DOCS_BUILD_ROOT` and the docs cache remain local; only the downloadable zip is affected (see above).

## AI Disclosure

This issue description was drafted with the assistance of a local model, Qwen 3.8 27B.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in pyproject.toml, requirements/prod.txt, and djangoproject/settings/prod.py to trace production storage configuration and its environment inputs. Then inspect releases/models.py, docs/management/commands/build_doc_release.py, and djangoproject/thumbnails.py, and run the existing test suite before adding coverage for the S3 path with a mocked backend. Done means local storage remains the default, S3 handles media, artifacts, docs zips, and thumbnails when enabled, and tests pass without AWS credentials.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, django, python
Domain
backend, cloud, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.