firebase / firebase/firebase-admin-python

Uploading files >100 MB to the Firebase Storage fails (Firebase Emulator Suite)

Abierto
#724 4 comentarios 0 reacciones 1 asignado Reclamado por @tonyjhuang Ver en GitHub
api: storage
Lenguaje dominante
Python
Estrellas
1.2k
Forks
359
Merge medio
5 d 6 min
PR fusionados (30 d)
2

Descripción

### Step 2: Describe your environment
* **Operating System version:** Linux Ubuntu 22.04.1
* **Firebase SDK version:** 12.5.4
* **Firebase Product:** Storage
* **Python version:** 3.10.12
* **Pip version:** 22.0.2

### Step 3: Describe the problem
I am using the `Firebase Emulator Suite - Storage` and `firebase-admin` in Python.

I wand to upload files to the `Firebase Emulator Suite - Storage` and I already set the system's environment variable `STORAGE_EMULATOR_HOST` to `http://localhost:9199`.

When I use the function `blob.upload_from_filename(filename=file_path)` and I upload a file that is smaller than the `_DEFAULT_CHUNKSIZE` (_100 MB_), which is defined in `blob.py`, everything is fine. But if my file exceeds the chunksize (_> 100 MB_), the file will not be uploaded correctly.

I tried to upload a 200 MB large file via the Firebase Emulator Suite UI in my browser (`http://127.0.0.1:4000/storage/`) and it worked. So the problem is not the emulator itsself.

These are the files I want to upload, which are larger than 700 MB per file.
![image](https://github.com/firebase/firebase-admin-python/assets/49234386/6b89e224-163c-426c-91e8-d076e4545607)

This is the UI after uploading two of the files. The maximum file size I can reach is 104.86 MB...
![image](https://github.com/firebase/firebase-admin-python/assets/49234386/e656a570-379f-4641-87df-80d026c4eb96)

The `Firebase Emulator Suite` does not throw any error, neither my code does.

#### Relevant Code:

In the following code I am creating the Firebase app object in the `__init__` function and use the storage bucket to upload files using the function `upload_from_filename()` in my custom function `upload_zip_from_path()`:
```
def __init__(self, app_name: str = "[DEFAULT]", config_yml_path: str = "config.yml"):
configs = FileUtils.load_config_yml(config_yml_path)
self.DEBUG = bool(configs["environment"]["debug"])
self.DATABASE_URL = configs["database"]["url"]
self.BUCKET_URL = configs["storage"]["url"]
token_file_path = configs["database"]["token_file_path"]

# Set host to local Firebase Emulator Suite Storage
if self.DEBUG:
os.environ["STORAGE_EMULATOR_HOST"] = "http://localhost:9199"

try:
self.APP_OBJ = firebase_admin.get_app(app_name)
except ValueError:
# If app does not exist, a value error is thrown
if len(token_file_path) > 0:
# Production: Authentication token needed
token = credentials.Certificate(token_file_path)
self.APP_OBJ = firebase_admin.initialize_app(
credential=token,
options={
"databaseURL": self.DATABASE_URL,
"storageBucket": self.BUCKET_URL,
},
name=app_name,
)
else:
# Development: No token needed using Firebase Emulator Suite
self.APP_OBJ = firebase_admin.initialize_app(
options={
"databaseURL": self.DATABASE_URL,
"storageBucket": self.BUCKET_URL,
},
name=app_name,
)
# References
self.SID_ROOT = db.reference("sid")
self.USER_ROOT = db.reference("user")
# Firebase app specific
self.APP_NAME = app_name
self.BUCKET = storage.bucket()

def upload_zip_from_path(self, zip_path:str):
# Set file name
if "/" in zip_path:
zip_name = zip_path.split("/")[-1] # "test/abc.zip" -> ["test", "abc.zip"] -> "abc.zip"
else:
zip_name = zip_path
# Get a reference to the Firebase Storage bucket
zip_folder = self.BUCKET.blob(f"zip/{zip_name}")
zip_folder.upload_from_filename(zip_path)
return zip_folder.public_url
```

On the other hand I tried to chunk the file by myself, but this did not went well.
With the following code the current chunk will overwrite the last chunk - this is not wanted... I wanted to append it to the file.
```
def resumable_upload(self, file_path, chunk_size=25 * 1024 * 1024):
# Set file name
if "/" in file_path:
destination_blob_name = file_path.split("/")[-1] # "test/abc.zip" -> ["test", "abc.zip"] -> "abc.zip"
else:
destination_blob_name = file_path

# Get a reference to the Firebase Storage bucket
bucket = storage.bucket()

# Create a resumable upload session
blob = bucket.blob(destination_blob_name)
blob.chunk_size = chunk_size
chunk_count = 0
chunk_start = 0
total_size = os.path.getsize(file_path)

with open(file_path, 'rb') as file:
while chunk_start < total_size:
chunk_end = min(chunk_start + chunk_size, total_size)
chunk_data = file.read(chunk_size)
blob.upload_from_string(chunk_data, content_type="application/zip")

chunk_start = chunk_end
chunk_count += 1
print(f'Uploaded chunk {chunk_count} / {total_size // chunk_size} ({chunk_start}/{total_size} bytes)')

print('Resumable upload completed.')
return blob.public_url
```

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.