langgenius / langgenius/dify

Add retention policy and quota controls for workflow/API uploaded files in upload_files storage

Open
#38,515 1 comment 1 reaction 0 assignees View on GitHub
project#dify
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [x] I have searched for existing issues and discussions about `upload_files`, storage cleanup, and orphaned files.
- [x] I have checked PR #18835. It helps with orphaned records/files, but does not address this use case because these files are still valid `upload_files` records and may still be referenced by workflow/message logs.

### Dify version

Self-hosted Docker deployment, Dify v1.13.x.

### Problem

In a self-hosted production deployment, workflow/API file uploads can grow `/app/api/storage/upload_files` indefinitely. Upload size and batch limits only reduce the growth rate; they do not provide any lifecycle management for accumulated files.

A real production example:

- About 480 users in one workspace.
- `api` and `worker` share the same `/app/api/storage` volume.
- One workflow app is called repeatedly through API and uploads PNG images.
- After a few months, `/app/api/storage/upload_files/` grew to about 433 GB.
- Database aggregation showed around 2,086,867 `image/png` records in `upload_files`, totaling about 406 GB.
- These records had `created_by_role = 'end_user'` and mapped back to a workflow end user for the same workflow app.

Example diagnostics:

```sql
SELECT extension, mime_type, COUNT(*) AS files,
pg_size_pretty(SUM(size)::bigint) AS total
FROM upload_files
GROUP BY extension, mime_type
ORDER BY SUM(size) DESC
LIMIT 30;
```

```sql
SELECT created_by_role, created_by,
COUNT(*) AS files,
pg_size_pretty(SUM(size)::bigint) AS total
FROM upload_files
WHERE mime_type = 'image/png'
AND key LIKE 'upload_files/%'
GROUP BY created_by_role, created_by
ORDER BY SUM(size) DESC
LIMIT 50;
```

### Current behavior

Dify currently provides several cleanup-related mechanisms, but none of them safely handles this case:

- `ENABLE_CLEAN_MESSAGES` / `flask clean-expired-messages` cleans message-related database records, but does not delete ordinary `upload_files` physical files.
- `WORKFLOW_LOG_CLEANUP_ENABLED` / workflow run cleanup removes workflow logs and related DB rows, but does not provide lifecycle management for uploaded files.
- `flask clear-orphaned-file-records` only removes unreferenced DB records.
- `flask remove-orphaned-files-on-storage` only removes physical files that are already absent from `upload_files` / `tool_files` records.
- PR #18835 is useful for orphan cleanup, but it is not a retention policy for valid uploaded files and it is global/high-load rather than scoped to an app/workflow.

As a result, valid workflow/API uploads can keep accumulating until the disk is full, even when each individual upload respects file size and batch limits.

### Expected behavior

Self-hosted administrators should have a supported way to control the lifecycle and growth of workflow/API uploaded files.

Possible capabilities:

1. Configurable retention policy for uploaded files, scoped by workspace, app, workflow, end user, MIME type, or storage prefix.
2. Per-workspace/app/workflow quotas for uploaded file count and total bytes.
3. A safe cleanup task that supports dry-run, batch size, rate limiting, and audit logs.
4. Clear distinction between temporary workflow/API uploads and permanent business files.
5. Optional Celery beat task to delete expired uploaded files by policy.
6. Storage usage reporting by workspace/app/workflow/end user/MIME type.
7. Behavior for quota exceeded: reject new uploads with a clear error before disk usage impacts the whole instance.

### Why this matters

For self-hosted deployments, `/app/api/storage/upload_files` is often backed by a local disk or a shared persistent volume. A single high-frequency workflow can consume hundreds of GB over time and affect the whole Dify instance.

This is not solved by per-request upload limits, because many valid API calls can still create unbounded accumulated storage. It also cannot be safely solved by generic orphan cleanup, because the files may still have valid DB records and references.

### Suggested direction

Add a first-class `upload_files` lifecycle management mechanism, for example:

- Store enough ownership metadata to trace uploaded files to workspace/app/workflow/run/end_user.
- Add a retention policy model/config for workflow/API uploads.
- Run a scheduled cleanup task that:
- selects candidate `upload_files` by scope and age;
- verifies references according to the selected policy;
- deletes physical files through the storage abstraction;
- deletes or marks the corresponding DB rows;
- records metrics/audit output;
- supports dry-run and small batches.

This would give self-hosted operators a safe and supported path to prevent `upload_files` from growing without bound.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing `flask clean-expired-messages`, `flask clear-orphaned-file-records`, and `flask remove-orphaned-files-on-storage` entry points, then compare their behavior with PR #18835. Trace how `upload_files` records and physical storage files are related. Done requires an agreed, supported retention or quota design that safely handles valid references, batching, dry runs, and cleanup reporting.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, flask, postgresql, python
Domain
backend, databases, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.