firebase / firebase/firebase-admin-python
Uploading files >100 MB to the Firebase Storage fails (Firebase Emulator Suite)
- Lenguaje dominante
- Python
- Estrellas
- 1.2k
- Forks
- 359
- Merge medio
- 5 d 6 min
- PR fusionados (30 d)
- 2
Descripción
### Step 2: Describe your environment
* **Operating System version:** Linux Ubuntu 22.04.1
* **Firebase SDK version:** 12.5.4
* **Firebase Product:** Storage
* **Python version:** 3.10.12
* **Pip version:** 22.0.2
### Step 3: Describe the problem
I am using the `Firebase Emulator Suite - Storage` and `firebase-admin` in Python.
I wand to upload files to the `Firebase Emulator Suite - Storage` and I already set the system's environment variable `STORAGE_EMULATOR_HOST` to `http://localhost:9199`.
When I use the function `blob.upload_from_filename(filename=file_path)` and I upload a file that is smaller than the `_DEFAULT_CHUNKSIZE` (_100 MB_), which is defined in `blob.py`, everything is fine. But if my file exceeds the chunksize (_> 100 MB_), the file will not be uploaded correctly.
I tried to upload a 200 MB large file via the Firebase Emulator Suite UI in my browser (`http://127.0.0.1:4000/storage/`) and it worked. So the problem is not the emulator itsself.
These are the files I want to upload, which are larger than 700 MB per file.

This is the UI after uploading two of the files. The maximum file size I can reach is 104.86 MB...

The `Firebase Emulator Suite` does not throw any error, neither my code does.
#### Relevant Code:
In the following code I am creating the Firebase app object in the `__init__` function and use the storage bucket to upload files using the function `upload_from_filename()` in my custom function `upload_zip_from_path()`:
```
def __init__(self, app_name: str = "[DEFAULT]", config_yml_path: str = "config.yml"):
configs = FileUtils.load_config_yml(config_yml_path)
self.DEBUG = bool(configs["environment"]["debug"])
self.DATABASE_URL = configs["database"]["url"]
self.BUCKET_URL = configs["storage"]["url"]
token_file_path = configs["database"]["token_file_path"]
# Set host to local Firebase Emulator Suite Storage
if self.DEBUG:
os.environ["STORAGE_EMULATOR_HOST"] = "http://localhost:9199"
try:
self.APP_OBJ = firebase_admin.get_app(app_name)
except ValueError:
# If app does not exist, a value error is thrown
if len(token_file_path) > 0:
# Production: Authentication token needed
token = credentials.Certificate(token_file_path)
self.APP_OBJ = firebase_admin.initialize_app(
credential=token,
options={
"databaseURL": self.DATABASE_URL,
"storageBucket": self.BUCKET_URL,
},
name=app_name,
)
else:
# Development: No token needed using Firebase Emulator Suite
self.APP_OBJ = firebase_admin.initialize_app(
options={
"databaseURL": self.DATABASE_URL,
"storageBucket": self.BUCKET_URL,
},
name=app_name,
)
# References
self.SID_ROOT = db.reference("sid")
self.USER_ROOT = db.reference("user")
# Firebase app specific
self.APP_NAME = app_name
self.BUCKET = storage.bucket()
def upload_zip_from_path(self, zip_path:str):
# Set file name
if "/" in zip_path:
zip_name = zip_path.split("/")[-1] # "test/abc.zip" -> ["test", "abc.zip"] -> "abc.zip"
else:
zip_name = zip_path
# Get a reference to the Firebase Storage bucket
zip_folder = self.BUCKET.blob(f"zip/{zip_name}")
zip_folder.upload_from_filename(zip_path)
return zip_folder.public_url
```
On the other hand I tried to chunk the file by myself, but this did not went well.
With the following code the current chunk will overwrite the last chunk - this is not wanted... I wanted to append it to the file.
```
def resumable_upload(self, file_path, chunk_size=25 * 1024 * 1024):
# Set file name
if "/" in file_path:
destination_blob_name = file_path.split("/")[-1] # "test/abc.zip" -> ["test", "abc.zip"] -> "abc.zip"
else:
destination_blob_name = file_path
# Get a reference to the Firebase Storage bucket
bucket = storage.bucket()
# Create a resumable upload session
blob = bucket.blob(destination_blob_name)
blob.chunk_size = chunk_size
chunk_count = 0
chunk_start = 0
total_size = os.path.getsize(file_path)
with open(file_path, 'rb') as file:
while chunk_start < total_size:
chunk_end = min(chunk_start + chunk_size, total_size)
chunk_data = file.read(chunk_size)
blob.upload_from_string(chunk_data, content_type="application/zip")
chunk_start = chunk_end
chunk_count += 1
print(f'Uploaded chunk {chunk_count} / {total_size // chunk_size} ({chunk_start}/{total_size} bytes)')
print('Resumable upload completed.')
return blob.public_url
```
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.