GDPR-compliant retention and deletion for workflow/API uploaded files
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] Please do not modify this template :) and fill in all the required fields.
### 1. Is this request related to a challenge you're experiencing? Tell me about your story.
We are running Dify v1.16.0 as a self-hosted Docker deployment.
Our workflows process uploaded audio files, documents and images through the API. These files may contain personal data and must therefore be deleted after a defined retention period for GDPR compliance.
Currently, Dify provides upload and download/preview functionality, but no supported API endpoint to delete an uploaded file by its `upload_file_id`.
The `FILES_ACCESS_TIMEOUT` setting only invalidates signed URLs. It does not delete the physical file or its database metadata.
As a result, uploaded files accumulate in the Dify storage over time. This creates both a storage management problem and a GDPR compliance risk, because personal data may remain stored indefinitely.
### 2. Additional context or comments
Dify already provides the following administrative cleanup commands:
- `flask clear-orphaned-file-records`
- `flask remove-orphaned-files-on-storage`
These commands were introduced in PR #18835:
https://github.com/langgenius/dify/pull/18835
However, they only remove files that are already orphaned. They do not provide a retention policy for valid files that are still referenced by workflow executions, messages or execution logs.
Related issues:
- #38515 - Add retention policy and quota controls for workflow/API uploaded files:
https://github.com/langgenius/dify/issues/38515
- #27420 - Add API and GUI cleanup mechanism to delete uploaded files:
https://github.com/langgenius/dify/issues/27420
- #19528 - Delete workflow files:
https://github.com/langgenius/dify/issues/19528
We need a supported and documented GDPR-compliant cleanup mechanism with the following capabilities:
- Configurable retention period for Workflow/API uploaded files
- Automatic scheduled deletion of expired files
- Optional deletion immediately after successful workflow execution
- Manual administrative cleanup command or API
- Deletion of both the physical storage object and the database metadata
- Safe handling of files referenced by messages or workflow execution logs
- Dry-run mode, audit logging and retry handling
- Clear separation between temporary workflow uploads and permanent resources such as knowledge-base documents, app icons and avatars
A public endpoint such as the following would also be useful:
`DELETE /v1/files/{file_id}`
Invalidating a signed URL or removing only an orphaned database record is not sufficient for GDPR compliance. The actual stored file and all related personal-data metadata must be removed.
### 3. Can you help us with this feature?
- [x] I am interested in contributing to this feature.
Contributor guide
Research direction
Start with the existing `flask clear-orphaned-file-records` and `flask remove-orphaned-files-on-storage` administrative commands, then review related issues #38515, #27420, and #19528. Define the scope for workflow/API uploads separately from permanent resources, including scheduled or manual cleanup, storage objects and database metadata, references, dry runs, audit logs, and retries; done means the retention behavior is documented and safely verifiable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, flask, python
- Domain
- api, backend, databases, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100