Cloud-CV / Cloud-CV/EvalAI

🗂️ Implement Submission & Logs Retention Policy Post-Challenge (1-Month Retention)

Open
#4,682 0 comments 0 reactions 1 assignee Claimed by @Zahed-Riyaz View on GitHub
Dominant language
Python
Stars
2k
Forks
984
Avg merge
2h 54m
Merged PRs (30d)
14

Description

**Description:**

To improve data storage efficiency and reduce long-term infrastructure costs, we propose implementing a **retention policy** for **submission files and logs** after a challenge has ended on EvalAI.

### 🎯 Objective

Once a challenge has ended, **submission metadata should be preserved**, but the following associated artifacts should be **retained for only 1 month**:

* Submission input files (e.g., prediction files, zip uploads)
* Submission output files (e.g., logs, result archives, stdout/stderr logs)

After the 1-month grace period, the storage used by these large artifacts should be released automatically.

---

### 🛠️ Proposed Implementation

#### 1. **Retention Trigger**

* Trigger based on the `end_date` of the challenge phase.
* Start countdown only if the `challenge_phase.is_public` is `False` (i.e., not accepting any more submissions).

#### 2. **Scheduled Cleanup**

* A periodic background job (e.g., via Celery beat or AWS Lambda/CloudWatch Events) that:

* Identifies expired submissions.
* Deletes associated large files from storage (AWS S3 or others).
* Keeps submission records in the database for audit/reference.

#### 3. **Audit Trail / Flags**

* Add a flag like `is_artifact_deleted` to each submission to avoid repeated deletion attempts.
* Log the deletion with timestamps for traceability.

#### 4. **Notification (Optional Enhancement)**

* Notify challenge hosts 14 days before deletion starts via email.
* Allow manual override if a host wants to retain data longer.

---

### 🔐 Considerations

* Ensure challenge host consent is accounted for — either as a configurable retention policy per challenge or platform-wide with clear documentation.
* Submissions from **host users** or **evaluation script outputs** may need a different policy if used for benchmarking or demo purposes.

---

### 📌 Related Work / References
Ref: https://github.com/Cloud-CV/EvalAI/pull/4586

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.