dlt-hub / dlt-hub/dlt

[docs] Google cloud run volume mounting

Open
#2,133 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### Documentation description

I run DLT in google cloudrun and have noticed when I load big tables it can get OOM even if it writes to files, as cloudrun doesn't have any "real" storage. What I've been doing instead is mounting a storage bucket and using pipeline_dir to direct the pipeline to use that as the directory. This seems to work well for me in the cases I've tested. But I've also seen that there are limitations with mounting a storage bucket as a directory, listed here https://cloud.google.com/run/docs/configuring/jobs/cloud-storage-volume-mounts . It would be good to have someone who knows how DLT works under the hood take a look at this and see if these limitations might cause issues (For example if two or more processes/ threads would write to the same file etc). If the limitations wouldn't cause issues I think it would be nice to include a section about it here
https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deploy-with-google-cloud-run
to help other in the future.

### Are you a dlt user?

Yes, I run dlt in production.

-----
Note from @rudolfix
Let's document finding below

Contributor guide

Open the contributing guide

Research direction

Start with the deploy-with-google-cloud-run guide and the linked Cloud Storage volume-mount limitations, then trace how pipeline_dir is used when loading large tables. Check whether concurrent processes or threads can write to the same file under these mounts. Done means documenting the verified limitations and safe usage in the deployment guide, or recording why no guidance should be added.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.