Issue when uploading large point cloud data

how the issue is created

1. create blank project from mergin desktop app
2. sync files to server
3. completed
4. add .las file to pc project folder
5. add file to qgis layers
6. wait for files to process .las.copc
7. once point cloud is displayed, save and sync project.

When trying to sync from the desktop (locally stored) 2 files, once it reaches 100% I get a 502 error message show, and the files fail to sync.

1. point cloud 2.86gb
2. copc.laz file 885mb

output from terminal

merginmaps-server  | [ACCESS] xxx.77.91.xxx, 192.168.1.233 POST /v1/project/push/chunk/f56ced65-a996-48da-93c5-d649203ff685/8a793045-6ade-480a-8b0c-5da64768a9ec  HTTP/1.0 200 72 - "Python-client/0.8.0 Plugin/2023.4.1 QGIS/3.34.2 (Windows/10)" 6797503 <15>
merginmaps-server  | /usr/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
merginmaps-server  |   return io.open(fd, *args, **kwargs)
merginmaps-server  | [CRITICAL] [11] WORKER TIMEOUT (pid:15)
merginmaps-db      | 2024-01-15 12:33:14.671 UTC [32] LOG:  unexpected EOF on client connection with an open transaction
merginmaps-server  | [INFO] [29] Booting worker with pid: 29
merginmaps-server  | [ACCESS] xxx.77.91.xxx, 192.168.1.233 POST /v1/project/push/cancel/f56ced65-a996-48da-93c5-d649203ff685  HTTP/1.0 200 0 - "Python-client/0.8.0 Plugin/2023.4.1 QGIS/3.34.2 (Windows/10)" 1641805 <16>
merginmaps-server  | [INFO] [29] Numpy was not imported, continuing without requires()
merginmaps-server  | [2024-01-15 12:33:39,448: INFO/MainProcess] Received task: mergin.stats.tasks.send_statistics[8dd201c9-3187-4d8d-a0d4-18e9f81926c0]
merginmaps-server  | [2024-01-15 12:33:39,492: INFO/ForkPoolWorker-1] Task mergin.stats.tasks.send_statistics[8dd201c9-3187-4d8d-a0d4-18e9f81926c0] succeeded in 0.004696418000094127s: None

after docker restarted, and database env set to DB_POOL_TIMEOUT=3000

merginmaps-server  | [ACCESS] xxx.77.91.xxx, 192.168.1.233 POST /v1/project/push/chunk/eaaab3f3-8eed-421d-a275-100483823d82/f7a09288-f043-400d-a2f5-58af4f75742d  HTTP/1.0 200 72 - "Python-client/0.8.0 Plugin/2023.4.1 QGIS/3.34.2 (Windows/10)" 118892 <15>
merginmaps-server  | /usr/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
merginmaps-server  |   return io.open(fd, *args, **kwargs)
merginmaps-server  | [CRITICAL] [11] WORKER TIMEOUT (pid:16)
merginmaps-server  | [INFO] [16] worker received SIGABRT signal
merginmaps-server  | [INFO] [16]
merginmaps-server  | # Thread: (140363629020992)
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/gevent/baseserver.py", line 34, in _handle_and_close_when_done
merginmaps-server  |   return handle(*args_tuple)
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/ggevent.py", line 155, in handle
merginmaps-server  |   super(GeventWorker, self).handle(listener, client, addr)
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
merginmaps-server  |   self.handle_request(listener_name, req, client, addr)
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/ggevent.py", line 159, in handle_request
merginmaps-server  |   super(GeventWorker, self).handle_request(listener_name, req, sock,
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/base_async.py", line 107, in handle_request
merginmaps-server  |   respiter = self.wsgi(environ, resp.start_response)
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2548, in __call__
merginmaps-server  |   return self.wsgi_app(environ, start_response)
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2525, in wsgi_app
merginmaps-server  |   response = self.full_dispatch_request()
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request
merginmaps-server  |   rv = self.dispatch_request()
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request
merginmaps-server  |   return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/connexion/decorators/decorator.py", line 68, in wrapper
merginmaps-server  |   response = function(request)
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/connexion/decorators/uri_parsing.py", line 149, in wrapper
merginmaps-server  |   response = function(request)
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/connexion/decorators/validation.py", line 399, in wrapper
merginmaps-server  |   return function(request)
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/connexion/decorators/response.py", line 112, in wrapper
merginmaps-server  |   response = function(request)
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/connexion/decorators/parameter.py", line 120, in wrapper
merginmaps-server  |   return function(**kwargs)
merginmaps-server  | File: "/app/mergin/auth/app.py", line 73, in wrapped_func
merginmaps-server  |   return f(*args, **kwargs)
merginmaps-server  | File: "/app/mergin/sync/public_api_controller.py", line 670, in wrapper
merginmaps-server  |   return f(*args, **kwargs)
merginmaps-server  | File: "/app/mergin/sync/public_api_controller.py", line 967, in push_finish
merginmaps-server  |   data = src.read(8192)
merginmaps-server  | File: "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/base.py", line 195, in handle_abort
merginmaps-server  |   self.cfg.worker_abort(self)
merginmaps-server  | File: "config.py", line 93, in worker_abort
merginmaps-server  |   for filename, lineno, name, line in traceback.extract_stack(stack):
merginmaps-server  | [ACCESS] xxx.77.91.xxx4, 192.168.1.233 POST /v1/project/push/cancel/eaaab3f3-8eed-421d-a275-100483823d82  HTTP/1.0 200 0 - "Python-client/0.8.0 Plugin/2023.4.1 QGIS/3.34.2 (Windows/10)" 227901 <15>
merginmaps-server  | [INFO] [16] Worker exiting (pid: 16)
merginmaps-server  | [INFO] [24] Booting worker with pid: 24
merginmaps-server  | [INFO] [24] Numpy was not imported, continuing without requires()

error.log

2024/01/15 13:34:52 [error] 29#29: *797 upstream prematurely closed connection while reading response header from upstream, client: 192.168.1.233, server: _, request: "POST /v1/project/push/finish/eaaab3f3-8eed-421d-a275-100483823d82 HT>

access.log

192.168.1.233 - - [15/Jan/2024:13:33:51 +0000] "POST /v1/project/push/chunk/eaaab3f3-8eed-421d-a275-100483823d82/f7a09288-f043-400d-a2f5-58af4f75742d HTTP/1.1" 200 72 "-" "Python-client/0.8.0 Plugin/2023.4.1 QGIS/3.34.2 (Windows/10)" "1>
192.168.1.233 - - [15/Jan/2024:13:34:52 +0000] "POST /v1/project/push/finish/eaaab3f3-8eed-421d-a275-100483823d82 HTTP/1.1" 502 157 "-" "Python-client/0.8.0 Plugin/2023.4.1 QGIS/3.34.2 (Windows/10)" "xxx.77.91.xxx"
192.168.1.233 - - [15/Jan/2024:13:34:52 +0000] "POST /v1/project/push/cancel/eaaab3f3-8eed-421d-a275-100483823d82 HTTP/1.1" 200 0 "-" "Python-client/0.8.0 Plugin/2023.4.1 QGIS/3.34.2 (Windows/10)" "xxx.77.91.xxx"
192.168.1.233 - - [15/Jan/2024:13:38:44 +0000] "GET /.well-known/sg-hosted-ping HTTP/1.1" 200 798 "-" "Avalon API Client" "35.213.196.135"

Start at mergin/sync/public_api_controller.py:967 in push_finish and review the logged POST /v1/project/push/finish path. Reproduce the sync with the reported large .las and .copc.laz files while watching the worker and nginx logs. Done means the upload finishes successfully without a 502, worker timeout, or forced cancellation.