element-hq / element-hq/synapse

`enable_local_media_storage: false` does not work properly

Open
#19,413 1 comment 2 reactions 0 assignees View on GitHub
A-Media-Repository
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

### Description

When `enable_local_media_storage` is set to `false`, `synapse-s3-storage-provider` and likely other storage providers are unable to `store_file` properly.

Storage provider's `store_file` method is only called with the `path` and `file_info` arguments. `path` is a relative path for the media relative to storage root, and `file_info` contains no information about where the file is stored, despite the comment saying otherwise.

https://github.com/element-hq/synapse/blob/53e8a3c14abd30a4499e7d4c0014e7233186ec65/synapse/media/storage_provider.py#L47-L54

In `synapse-s3-storage-provider`, the local media store path is used to construct the full file path from the relative path:

https://github.com/matrix-org/synapse-s3-storage-provider/blob/2dbba1020fe6fb84d79f1a23c925e508d9ba15e1/s3_storage_provider.py#L60

But this no longer works with the temporary files used when `enable_local_media_storage` is set to `false`

### Steps to reproduce

- configure synapse with synapse-s3-storage-provider
- set `enable_local_media_storage` to false
- try to send some media

### Homeserver

nichi.co

### Synapse Version

1.146.0

### Installation Method

Other (please mention below)

### Database

Single PostgreSQL server, not restored from backup

### Workers

Single process

### Platform

NixOS using the matrix-synapse module.

### Configuration

_No response_

### Relevant log output

```shell
Jan 27 19:49:13 hio0 synapse[402679]: synapse.http.server: [POST-12959] Failed handle request via 'UploadServlet':
Traceback (most recent call last):
File "/nix/store/kwiwps7k704bdjx17pb3b5b8vlwl04ky-matrix-synapse-1.146.0/lib/python3.13/site-packages/synapse/http/server.py", line 335, in _async_render_wrapper
callback_return = await self._async_render(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/kwiwps7k704bdjx17pb3b5b8vlwl04ky-matrix-synapse-1.146.0/lib/python3.13/site-packages/synapse/http/server.py", line 576, in _async_render
callback_return = await raw_callback_return
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/kwiwps7k704bdjx17pb3b5b8vlwl04ky-matrix-synapse-1.146.0/lib/python3.13/site-packages/synapse/rest/media/upload_resource.py", line 123, in on_POST
content_uri = await self.media_repo.create_or_update_content(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
media_type, upload_name, content, content_length, requester.user
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/kwiwps7k704bdjx17pb3b5b8vlwl04ky-matrix-synapse-1.146.0/lib/python3.13/site-packages/synapse/media/media_repository.py", line 346, in create_or_update_content
fname = await self.media_storage.store_file(sha256reader.wrap(), file_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
--
...<6 lines>...
)
^
File "/nix/store/kwiwps7k704bdjx17pb3b5b8vlwl04ky-matrix-synapse-1.146.0/lib/python3.13/site-packages/synapse/module_api/__init__.py", line 1737, in defer_to_threadpool
return await defer_to_threadpool(
^^^^^^^^^^^^^^^^^^^^^^^^^^
self._hs.get_reactor(), threadpool, f, *args, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/twisted/internet/defer.py", line 1187, in __iter__
yield self
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/twisted/python/threadpool.py", line 269, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/twisted/python/threadpool.py", line 285, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ctx, func, *args, **kw
^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
File "/nix/store/kwiwps7k704bdjx17pb3b5b8vlwl04ky-matrix-synapse-1.146.0/lib/python3.13/site-packages/synapse/logging/context.py", line 1207, in g
return f(*args, **kwargs)
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/botocore/context.py", line 123, in wrapper
return func(*args, **kwargs)
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/boto3/s3/inject.py", line 175, in upload_file
return transfer.upload_file(
~~~~~~~~~~~~~~~~~~~~^
filename=Filename,
^^^^^^^^^^^^^^^^^^
...<3 lines>...
callback=Callback,
^^^^^^^^^^^^^^^^^^
)
^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/boto3/s3/transfer.py", line 372, in upload_file
future.result()
~~~~~~~~~~~~~^^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/s3transfer/futures.py", line 111, in result
return self._coordinator.result()
~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/s3transfer/futures.py", line 287, in result
raise self._exception
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/s3transfer/tasks.py", line 272, in _main
self._submit(transfer_future=transfer_future, **kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/s3transfer/upload.py", line 596, in _submit
upload_input_manager.provide_transfer_size(transfer_future)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/s3transfer/upload.py", line 245, in provide_transfer_size
self._osutil.get_file_size(transfer_future.meta.call_args.fileobj)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/kdph64a34ldach4l3nqp7knjwiknnj78-python3-3.13.11-env/lib/python3.13/site-packages/s3transfer/utils.py", line 262, in get_file_size
return os.path.getsize(filename)
~~~~~~~~~~~~~~~^^^^^^^^^^
File "", line 86, in getsize
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/matrix-synapse/media_store/local_content/eK/oP/GDOeGnbVSffQTQjLOinp'
```

### Anything else that would be useful to know?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with synapse/media/storage_provider.py around lines 47-54 and the call from synapse/media/media_repository.py:346, then compare the storage-provider contract with synapse-s3-storage-provider/s3_storage_provider.py around line 60. Reproduce an upload with enable_local_media_storage set to false and verify that the provider receives usable file information and the media upload completes without FileNotFoundError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.