codalab / codalab/codabench

Storage Analytics never completes

Open
#1,661 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug
Dominant language
Python
Stars
176
Forks
74
Avg merge
5d 3h
Merged PRs (30d)
21

Description

Hello,
It seems like the storage analytics never completes once started, and might not even trigger the soft timeout ?

This is the result of docker compose logs --no-log-prefix site_worker | grep "analytics"

  . analytics.tasks.create_storage_analytics_snapshot
  . analytics.tasks.delete_orphan_files
  . analytics.tasks.reset_computed_storage_analytics
  . apps.analytics.tasks.create_storage_analytics_snapshot
  . apps.analytics.tasks.delete_orphan_files
  . apps.analytics.tasks.reset_computed_storage_analytics
  . analytics.tasks.create_storage_analytics_snapshot
  . analytics.tasks.delete_orphan_files
  . analytics.tasks.reset_computed_storage_analytics
  . apps.analytics.tasks.create_storage_analytics_snapshot
  . apps.analytics.tasks.delete_orphan_files
  . apps.analytics.tasks.reset_computed_storage_analytics
[2024-11-08 11:26:51,656: INFO/MainProcess] Received task: analytics.tasks.create_storage_analytics_snapshot[88f27753-478f-433b-8e5a-0b8e7cb7f854]  
[2024-11-08 11:26:51,661: INFO/ForkPoolWorker-2] Task create_storage_analytics_snapshot started
[2024-11-10 02:00:00,251: INFO/Beat] Scheduler: Sending due task create_storage_analytics_snapshot (analytics.tasks.create_storage_analytics_snapshot)
[2024-11-10 02:00:00,461: INFO/MainProcess] Received task: analytics.tasks.create_storage_analytics_snapshot[ad36d01a-efcd-4506-bda7-b6ae57362a17]  
[2024-11-10 02:00:00,464: INFO/ForkPoolWorker-11] Task create_storage_analytics_snapshot started
  . analytics.tasks.create_storage_analytics_snapshot
  . analytics.tasks.delete_orphan_files
  . analytics.tasks.reset_computed_storage_analytics
  . apps.analytics.tasks.create_storage_analytics_snapshot
  . apps.analytics.tasks.delete_orphan_files
  . apps.analytics.tasks.reset_computed_storage_analytics

The last successful analytics on the production server dates from September 2, 2024.

I launched the analytics on the test server, I will post the results here when it's done, but it does show that the last successful run was November 10, 2024.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the analytics.tasks.create_storage_analytics_snapshot task and its apps.analytics counterpart, then compare the worker and scheduler logs from docker compose. Check why runs remain at “started” without completing or triggering the soft timeout. Done means storage analytics completes successfully and a later scheduled run does not overlap or remain stuck.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.