HumanSignal / HumanSignal/label-studio

Sync failure from gcs

Open
#7,194 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
28.3k
Forks
3.7k
Avg merge
14h
Merged PRs (30d)
15

Description

**Description**
When creating a weekly task with 1,000–2,000 images sourced from a Google Cloud Storage (GCS) bucket, triggering the Sync Storage function results in an Internal Server Error. However, despite the error, the sync process eventually completes after multiple attempts.

**To Reproduce**

1. Open Label Studio.
2. Create a project and configure it to import images from a GCS storage bucket.
3. Add around 1,000 images to the folder.
4. Sync the storage using either of the following methods:
- Navigate to Settings and click Sync from Cloud Storage.
- Send a request to the API endpoint:
http://localhost:8080/api/storages/gcs/1/sync
5. After some time, an Internal Server Error appears.

**Expected behavior**

The sync process should complete successfully without an error.

**Possible Cause**
Based on a review of the code, the issue seems to originate from the _scan_and_create_links function. It iterates through all entries without optimization, which may be causing performance issues. Batch processing and threading could improve efficiency.

**Screenshots**

![Image](https://github.com/user-attachments/assets/3185b1e4-b39f-4bb5-a65d-9990c7705d42)

**Environment (please complete the following information):**
- OS: linux

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.