HumanSignal / HumanSignal/label-studio
Physical files not deleted when tasks are deleted (orphan files accumulate)
- Dominant language
- TypeScript
- Stars
- 28.3k
- Forks
- 3.7k
- Avg merge
- 14h
- Merged PRs (30d)
- 15
Description
## Problem Description
When a task is deleted in Label Studio, the associated `FileUpload` database record is removed, but the **physical file remains on disk** in the `media/upload/{project_id}/` directory. This creates orphan files that consume storage indefinitely and cannot be cleaned up through the Label Studio UI or API.
### Observed Behavior
| Action | Database Effect | Filesystem Effect |
|--------|-----------------|-------------------|
| Upload file to project | Creates `FileUpload` record | Writes file to `media/upload/{project_id}/` |
| Delete task referencing file | Deletes `Task` record | **No effect** (file remains) |
| Delete task | Deletes `FileUpload` record | **No effect** (file remains) |
### Impact
- Storage grows unbounded over time
- No built-in mechanism to identify or remove orphan files
- Users must manually locate and delete files from the filesystem
- In containerized deployments, orphans accumulate across restarts
### Environment
- **Label Studio Version:** 1.22.0
- **Deployment:** Docker (local volume mount)
## Reproduction Steps
1. Upload a file to a project via the UI or API
2. Verify file exists on disk: `ls media/upload/{project_id}/`
3. Delete the task that references the file
4. Verify the physical file still exists on disk (it will)
## Root Cause
Django's `FileField` does not automatically delete physical files when records are deleted - this is intentional Django behavior to prevent accidental data loss. However, Label Studio does not implement a `post_delete` signal handler to clean up files when `FileUpload` records are removed.
## Proposed Solutions
### Solution 1: Post-Delete Signal Handler (Recommended)
Add a Django signal handler to delete physical files when `FileUpload` records are deleted:
```python
from django.db.models.signals import pre_delete, post_delete
from django.dispatch import receiver
@receiver(pre_delete, sender=FileUpload)
def cache_file_path_before_delete(sender, instance, **kwargs):
"""Cache the file path before deletion."""
if instance.file:
instance._file_path_to_delete = instance.file.path
else:
instance._file_path_to_delete = None
@receiver(post_delete, sender=FileUpload)
def delete_file_on_fileupload_delete(sender, instance, **kwargs):
"""Delete the physical file after FileUpload record is deleted."""
file_path = getattr(instance, '_file_path_to_delete', None)
if file_path and os.path.isfile(file_path):
try:
os.remove(file_path)
logger.info(f"Deleted file: {file_path}")
except OSError as e:
logger.error(f"Failed to delete file {file_path}: {e}")
```
### Solution 2: Management Command
Add a `cleanup_orphan_files` management command for manual cleanup of existing orphans:
```bash
python manage.py cleanup_orphan_files --dry-run
python manage.py cleanup_orphan_files --project-id 4 --confirm
```
### Solution 3: API Endpoint
Add an admin-only API endpoint for programmatic cleanup:
```
GET /api/projects/{id}/orphan-files # List orphans
DELETE /api/projects/{id}/orphan-files?confirm=true # Delete orphans
```
## Additional Context
I've prepared a detailed investigation document with:
- Complete reproduction steps with curl commands
- Full implementation code for all three solutions
- Unit test examples
- A standalone Python script workaround for current users
Happy to share the full document or submit a PR if there's interest.
Contributor guide
Assessment
This issue has not been assessed yet.