HumanSignal / HumanSignal/label-studio

Physical files not deleted when tasks are deleted (orphan files accumulate)

Open
#9,305 2 comments 1 reaction 0 assignees View on GitHub
community_docs
Dominant language
TypeScript
Stars
28.3k
Forks
3.7k
Avg merge
14h
Merged PRs (30d)
15

Description

## Problem Description

When a task is deleted in Label Studio, the associated `FileUpload` database record is removed, but the **physical file remains on disk** in the `media/upload/{project_id}/` directory. This creates orphan files that consume storage indefinitely and cannot be cleaned up through the Label Studio UI or API.

### Observed Behavior

| Action | Database Effect | Filesystem Effect |
|--------|-----------------|-------------------|
| Upload file to project | Creates `FileUpload` record | Writes file to `media/upload/{project_id}/` |
| Delete task referencing file | Deletes `Task` record | **No effect** (file remains) |
| Delete task | Deletes `FileUpload` record | **No effect** (file remains) |

### Impact

- Storage grows unbounded over time
- No built-in mechanism to identify or remove orphan files
- Users must manually locate and delete files from the filesystem
- In containerized deployments, orphans accumulate across restarts

### Environment

- **Label Studio Version:** 1.22.0
- **Deployment:** Docker (local volume mount)

## Reproduction Steps

1. Upload a file to a project via the UI or API
2. Verify file exists on disk: `ls media/upload/{project_id}/`
3. Delete the task that references the file
4. Verify the physical file still exists on disk (it will)

## Root Cause

Django's `FileField` does not automatically delete physical files when records are deleted - this is intentional Django behavior to prevent accidental data loss. However, Label Studio does not implement a `post_delete` signal handler to clean up files when `FileUpload` records are removed.

## Proposed Solutions

### Solution 1: Post-Delete Signal Handler (Recommended)

Add a Django signal handler to delete physical files when `FileUpload` records are deleted:

```python
from django.db.models.signals import pre_delete, post_delete
from django.dispatch import receiver

@receiver(pre_delete, sender=FileUpload)
def cache_file_path_before_delete(sender, instance, **kwargs):
"""Cache the file path before deletion."""
if instance.file:
instance._file_path_to_delete = instance.file.path
else:
instance._file_path_to_delete = None

@receiver(post_delete, sender=FileUpload)
def delete_file_on_fileupload_delete(sender, instance, **kwargs):
"""Delete the physical file after FileUpload record is deleted."""
file_path = getattr(instance, '_file_path_to_delete', None)
if file_path and os.path.isfile(file_path):
try:
os.remove(file_path)
logger.info(f"Deleted file: {file_path}")
except OSError as e:
logger.error(f"Failed to delete file {file_path}: {e}")
```

### Solution 2: Management Command

Add a `cleanup_orphan_files` management command for manual cleanup of existing orphans:

```bash
python manage.py cleanup_orphan_files --dry-run
python manage.py cleanup_orphan_files --project-id 4 --confirm
```

### Solution 3: API Endpoint

Add an admin-only API endpoint for programmatic cleanup:

```
GET /api/projects/{id}/orphan-files # List orphans
DELETE /api/projects/{id}/orphan-files?confirm=true # Delete orphans
```

## Additional Context

I've prepared a detailed investigation document with:
- Complete reproduction steps with curl commands
- Full implementation code for all three solutions
- Unit test examples
- A standalone Python script workaround for current users

Happy to share the full document or submit a PR if there's interest.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.