langgenius / langgenius/dify

GDPR-compliant retention and deletion for workflow/API uploaded files

Open
#39,979 0 comments 2 reactions 0 assignees View on GitHub
💪 enhancement
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] Please do not modify this template :) and fill in all the required fields.

### 1. Is this request related to a challenge you're experiencing? Tell me about your story.

We are running Dify v1.16.0 as a self-hosted Docker deployment.

Our workflows process uploaded audio files, documents and images through the API. These files may contain personal data and must therefore be deleted after a defined retention period for GDPR compliance.

Currently, Dify provides upload and download/preview functionality, but no supported API endpoint to delete an uploaded file by its `upload_file_id`.

The `FILES_ACCESS_TIMEOUT` setting only invalidates signed URLs. It does not delete the physical file or its database metadata.

As a result, uploaded files accumulate in the Dify storage over time. This creates both a storage management problem and a GDPR compliance risk, because personal data may remain stored indefinitely.

### 2. Additional context or comments

Dify already provides the following administrative cleanup commands:

- `flask clear-orphaned-file-records`
- `flask remove-orphaned-files-on-storage`

These commands were introduced in PR #18835:

https://github.com/langgenius/dify/pull/18835

However, they only remove files that are already orphaned. They do not provide a retention policy for valid files that are still referenced by workflow executions, messages or execution logs.

Related issues:

- #38515 - Add retention policy and quota controls for workflow/API uploaded files:
https://github.com/langgenius/dify/issues/38515

- #27420 - Add API and GUI cleanup mechanism to delete uploaded files:
https://github.com/langgenius/dify/issues/27420

- #19528 - Delete workflow files:
https://github.com/langgenius/dify/issues/19528

We need a supported and documented GDPR-compliant cleanup mechanism with the following capabilities:

- Configurable retention period for Workflow/API uploaded files
- Automatic scheduled deletion of expired files
- Optional deletion immediately after successful workflow execution
- Manual administrative cleanup command or API
- Deletion of both the physical storage object and the database metadata
- Safe handling of files referenced by messages or workflow execution logs
- Dry-run mode, audit logging and retry handling
- Clear separation between temporary workflow uploads and permanent resources such as knowledge-base documents, app icons and avatars

A public endpoint such as the following would also be useful:

`DELETE /v1/files/{file_id}`

Invalidating a signed URL or removing only an orphaned database record is not sufficient for GDPR compliance. The actual stored file and all related personal-data metadata must be removed.

### 3. Can you help us with this feature?

- [x] I am interested in contributing to this feature.

Contributor guide

Open the contributing guide

Research direction

Start with the existing `flask clear-orphaned-file-records` and `flask remove-orphaned-files-on-storage` administrative commands, then review related issues #38515, #27420, and #19528. Define the scope for workflow/API uploads separately from permanent resources, including scheduled or manual cleanup, storage objects and database metadata, references, dry runs, audit logs, and retries; done means the retention behavior is documented and safely verifiable.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, flask, python
Domain
api, backend, databases, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.