firebase / firebase/firebase-admin-python

Uploading files >100 MB to the Firebase Storage fails (Firebase Emulator Suite)

Đang mở
#724 4 bình luận 0 reaction 1 người được giao Được @tonyjhuang nhận Xem trên GitHub
api: storage
Ngôn ngữ chính
Python
Star
1.2k
Fork
359
Merge trung bình
5 ngày 6 phút
Pull request đã merge (30 ngày)
2

Mô tả

### Step 2: Describe your environment
* **Operating System version:** Linux Ubuntu 22.04.1
* **Firebase SDK version:** 12.5.4
* **Firebase Product:** Storage
* **Python version:** 3.10.12
* **Pip version:** 22.0.2

### Step 3: Describe the problem
I am using the `Firebase Emulator Suite - Storage` and `firebase-admin` in Python.

I wand to upload files to the `Firebase Emulator Suite - Storage` and I already set the system's environment variable `STORAGE_EMULATOR_HOST` to `http://localhost:9199`.

When I use the function `blob.upload_from_filename(filename=file_path)` and I upload a file that is smaller than the `_DEFAULT_CHUNKSIZE` (_100 MB_), which is defined in `blob.py`, everything is fine. But if my file exceeds the chunksize (_> 100 MB_), the file will not be uploaded correctly.

I tried to upload a 200 MB large file via the Firebase Emulator Suite UI in my browser (`http://127.0.0.1:4000/storage/`) and it worked. So the problem is not the emulator itsself.

These are the files I want to upload, which are larger than 700 MB per file.
![image](https://github.com/firebase/firebase-admin-python/assets/49234386/6b89e224-163c-426c-91e8-d076e4545607)

This is the UI after uploading two of the files. The maximum file size I can reach is 104.86 MB...
![image](https://github.com/firebase/firebase-admin-python/assets/49234386/e656a570-379f-4641-87df-80d026c4eb96)

The `Firebase Emulator Suite` does not throw any error, neither my code does.

#### Relevant Code:

In the following code I am creating the Firebase app object in the `__init__` function and use the storage bucket to upload files using the function `upload_from_filename()` in my custom function `upload_zip_from_path()`:
```
def __init__(self, app_name: str = "[DEFAULT]", config_yml_path: str = "config.yml"):
configs = FileUtils.load_config_yml(config_yml_path)
self.DEBUG = bool(configs["environment"]["debug"])
self.DATABASE_URL = configs["database"]["url"]
self.BUCKET_URL = configs["storage"]["url"]
token_file_path = configs["database"]["token_file_path"]

# Set host to local Firebase Emulator Suite Storage
if self.DEBUG:
os.environ["STORAGE_EMULATOR_HOST"] = "http://localhost:9199"

try:
self.APP_OBJ = firebase_admin.get_app(app_name)
except ValueError:
# If app does not exist, a value error is thrown
if len(token_file_path) > 0:
# Production: Authentication token needed
token = credentials.Certificate(token_file_path)
self.APP_OBJ = firebase_admin.initialize_app(
credential=token,
options={
"databaseURL": self.DATABASE_URL,
"storageBucket": self.BUCKET_URL,
},
name=app_name,
)
else:
# Development: No token needed using Firebase Emulator Suite
self.APP_OBJ = firebase_admin.initialize_app(
options={
"databaseURL": self.DATABASE_URL,
"storageBucket": self.BUCKET_URL,
},
name=app_name,
)
# References
self.SID_ROOT = db.reference("sid")
self.USER_ROOT = db.reference("user")
# Firebase app specific
self.APP_NAME = app_name
self.BUCKET = storage.bucket()

def upload_zip_from_path(self, zip_path:str):
# Set file name
if "/" in zip_path:
zip_name = zip_path.split("/")[-1] # "test/abc.zip" -> ["test", "abc.zip"] -> "abc.zip"
else:
zip_name = zip_path
# Get a reference to the Firebase Storage bucket
zip_folder = self.BUCKET.blob(f"zip/{zip_name}")
zip_folder.upload_from_filename(zip_path)
return zip_folder.public_url
```

On the other hand I tried to chunk the file by myself, but this did not went well.
With the following code the current chunk will overwrite the last chunk - this is not wanted... I wanted to append it to the file.
```
def resumable_upload(self, file_path, chunk_size=25 * 1024 * 1024):
# Set file name
if "/" in file_path:
destination_blob_name = file_path.split("/")[-1] # "test/abc.zip" -> ["test", "abc.zip"] -> "abc.zip"
else:
destination_blob_name = file_path

# Get a reference to the Firebase Storage bucket
bucket = storage.bucket()

# Create a resumable upload session
blob = bucket.blob(destination_blob_name)
blob.chunk_size = chunk_size
chunk_count = 0
chunk_start = 0
total_size = os.path.getsize(file_path)

with open(file_path, 'rb') as file:
while chunk_start < total_size:
chunk_end = min(chunk_start + chunk_size, total_size)
chunk_data = file.read(chunk_size)
blob.upload_from_string(chunk_data, content_type="application/zip")

chunk_start = chunk_end
chunk_count += 1
print(f'Uploaded chunk {chunk_count} / {total_size // chunk_size} ({chunk_start}/{total_size} bytes)')

print('Resumable upload completed.')
return blob.public_url
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.