element-hq / element-hq/synapse
`enable_local_media_storage: false` does not work properly
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
### Description
When `enable_local_media_storage` is set to `false`, `synapse-s3-storage-provider` and likely other storage providers are unable to `store_file` properly.
Storage provider's `store_file` method is only called with the `path` and `file_info` arguments. `path` is a relative path for the media relative to storage root, and `file_info` contains no information about where the file is stored, despite the comment saying otherwise.
https://github.com/element-hq/synapse/blob/53e8a3c14abd30a4499e7d4c0014e7233186ec65/synapse/media/storage_provider.py#L47-L54
In `synapse-s3-storage-provider`, the local media store path is used to construct the full file path from the relative path:
https://github.com/matrix-org/synapse-s3-storage-provider/blob/2dbba1020fe6fb84d79f1a23c925e508d9ba15e1/s3_storage_provider.py#L60
But this no longer works with the temporary files used when `enable_local_media_storage` is set to `false`
### Steps to reproduce
- configure synapse with synapse-s3-storage-provider
- set `enable_local_media_storage` to false
- try to send some media
### Homeserver
nichi.co
### Synapse Version
1.146.0
### Installation Method
Other (please mention below)
### Database
Single PostgreSQL server, not restored from backup
### Workers
Single process
### Platform
NixOS using the matrix-synapse module.
### Configuration
_No response_
### Relevant log output
```shell
Jan 27 19:49:13 hio0 synapse[402679]: synapse.http.server: [POST-12959] Failed handle request via 'UploadServlet':
Traceback (most recent call last):
File "/nix/store/kwiwps7k704bdjx17pb3b5b8vlwl04ky-matrix-synapse-1.146.0/lib/python3.13/site-packages/synapse/http/server.py", line 335, in _async_render_wrapper
callback_return = await self._async_render(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/kwiwps7k704bdjx17pb3b5b8vlwl04ky-matrix-synapse-1.146.0/lib/python3.13/site-packages/synapse/http/server.py", line 576, in _async_render
callback_return = await raw_callback_return
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/kwiwps7k704bdjx17pb3b5b8vlwl04ky-matrix-synapse-1.146.0/lib/python3.13/site-packages/synapse/rest/media/upload_resource.py", line 123, in on_POST
content_uri = await self.media_repo.create_or_update_content(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
media_type, upload_name, content, content_length, requester.user
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/kwiwps7k704bdjx17pb3b5b8vlwl04ky-matrix-synapse-1.146.0/lib/python3.13/site-packages/synapse/media/media_repository.py", line 346, in create_or_update_content
fname = await self.media_storage.store_file(sha256reader.wrap(), file_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
--
...<6 lines>...
)
^
File "/nix/store/kwiwps7k704bdjx17pb3b5b8vlwl04ky-matrix-synapse-1.146.0/lib/python3.13/site-packages/synapse/module_api/__init__.py", line 1737, in defer_to_threadpool
return await defer_to_threadpool(
^^^^^^^^^^^^^^^^^^^^^^^^^^
self._hs.get_reactor(), threadpool, f, *args, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/twisted/internet/defer.py", line 1187, in __iter__
yield self
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/twisted/python/threadpool.py", line 269, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/twisted/python/threadpool.py", line 285, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ctx, func, *args, **kw
^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
File "/nix/store/kwiwps7k704bdjx17pb3b5b8vlwl04ky-matrix-synapse-1.146.0/lib/python3.13/site-packages/synapse/logging/context.py", line 1207, in g
return f(*args, **kwargs)
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/botocore/context.py", line 123, in wrapper
return func(*args, **kwargs)
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/boto3/s3/inject.py", line 175, in upload_file
return transfer.upload_file(
~~~~~~~~~~~~~~~~~~~~^
filename=Filename,
^^^^^^^^^^^^^^^^^^
...<3 lines>...
callback=Callback,
^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/boto3/s3/transfer.py", line 372, in upload_file
future.result()
~~~~~~~~~~~~~^^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/s3transfer/futures.py", line 111, in result
return self._coordinator.result()
~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/s3transfer/futures.py", line 287, in result
raise self._exception
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/s3transfer/tasks.py", line 272, in _main
self._submit(transfer_future=transfer_future, **kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/s3transfer/upload.py", line 596, in _submit
upload_input_manager.provide_transfer_size(transfer_future)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/s3transfer/upload.py", line 245, in provide_transfer_size
self._osutil.get_file_size(transfer_future.meta.call_args.fileobj)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/s3transfer/utils.py", line 262, in get_file_size
return os.path.getsize(filename)
~~~~~~~~~~~~~~~^^^^^^^^^^
File "", line 86, in getsize
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/matrix-synapse/media_store/local_content/eK/oP/GDOeGnbVSffQTQjLOinp'
```
### Anything else that would be useful to know?
_No response_
Contributor guide
Research direction
Start with synapse/media/storage_provider.py around lines 47-54 and the call from synapse/media/media_repository.py:346, then compare the storage-provider contract with synapse-s3-storage-provider/s3_storage_provider.py around line 60. Reproduce an upload with enable_local_media_storage set to false and verify that the provider receives usable file information and the media upload completes without FileNotFoundError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100