HumanSignal / HumanSignal/label-studio
Request for Enhancements to External Storage Sync Functionality
- Dominant language
- TypeScript
- Stars
- 28.3k
- Forks
- 3.7k
- Avg merge
- 14h
- Merged PRs (30d)
- 15
Description
Hello,
I hope this message finds you well.
I am reaching out to inquire if there are any plans to enhance the external storage synchronization functionality for local storage. Currently, the API call initiates a one-time sync for the entire project. While this works for smaller projects, it becomes challenging for larger ones (e.g., those with more than 5,000 or so tasks), as the sync process often times out before completion.
Although I have implemented a workaround by extending the timeout period #5890, this approach becomes increasingly impractical as project size continues to grow. For instance, I have a project with over 60,000 tasks, and the synchronization process takes more than 20 minutes to complete.
A potential improvement could be the introduction of multi-threaded synchronization. Additionally, providing users with more granular control over the synchronization process would be highly beneficial. For example, having the ability to sync specific tasks based on criteria could greatly enhance usability. My top two suggestions for such functionality include:
1. Synchronizing tasks in order of priority (e.g., oldest or newest tasks first).
2. Synchronizing tasks by a specified range of task IDs.
Would such enhancements be feasible?
Thank you for your time and consideration. I look forward to your thoughts.
Best regards,
Willie
Contributor guide
Assessment
This issue has not been assessed yet.