deepcelllineage / deepcelllineage/mitolin
Create an architecture for open source data exchange
- Dominant language
- Jupyter Notebook
- Stars
- 2
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
## Aim
Design a straight forward, low barrier to entry, solution for the movement of data between open source collaborators of the DCL project.
## Background
We have a number of different systems that we are trying to make work together
- Individual's laptops
- [UCI hpc](https://hpc.oit.uci.edu/)
- Serge's GPU
- Google Drive
Data sizes vary. We want to be able to access and share data from 10s, 100s, and 1000s of files that can be as small as 20KB to as large as 10GB. Larger files can be processed on the HPC. We want a system to share small and medium sized files ASAP, which we can build on as our analyses get more data hungry.
### UCI HPC related docs
1. [How to transfer large amounts of data](http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html)
2. [OIT Transfering data](https://hpc.oit.uci.edu/transferring-data)
1. see examples
3. UCI Cloud File Storage Options
1. Google drive
1. [setup for UCI staff](https://www.oit.uci.edu/help/google/g-suite-setup/)
2. make folders public
4. [UCI Webfiles](https://www.oit.uci.edu/webfiles/)
1. not really what I want
2. How to set up Private and Public Directories in WebFiles v7: [UCI tutorial](https://uci.service-now.com/kb_view.do?sysparm_article=KB0010526)
3. upload files [UCI tutorial](https://uci.service-now.com/kb_view.do?sysparm_article=KB0010527)
## Questions?
Please put questions related to this issue in this issue thread.
If you want a quick response, post a link to your comment in this thread to Slack #deepcelllineage and tag @Deena or others. You can also DM @Deena.
To join Slack enter your email address [here](http://bit.ly/JoinSlackFastaiSFbay). For questions NOT specifically related to this issue, get in touch through any of the communication methods listed in [DCL's overview README](https://github.com/deepcelllineage/overview/blob/master/README.md).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the linked UCI HPC transfer, Google Drive, and WebFiles documentation, then compare those options against the stated file-size and collaboration requirements. Done means documenting a low-barrier data-exchange architecture that supports laptops, HPC, GPU, and shared storage, with a clear path from small files to larger datasets.
Written by the indexing model from the issue text.
Assessment
- Domain
- cloud, data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100