Cannot leave/remove deleted room

**Describe the bug**

After a room was removed using [`POST /_synapse/admin/v1/rooms//delete` (`purge=True`)](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/rooms.md#delete-room-api) and all users were moved to a new room, the old room still shows up in Element and users cannot leave or remove the room. You just get an error that the rom is unknown:

**To Reproduce**

1. Create a room
2. Use [`POST /_synapse/admin/v1/rooms//delete` (`purge=True`)](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/rooms.md#delete-room-api) to delete the room
3. Search for room in Element (it is still there)
4. Try leaving the room

**Expected behavior**

If the room does not exist, I would expect that it will automatically be removed (best case) but I should at least be able to leave/remove the room manually from the app regardless of it existing on the server.

**Screenshots**

https://user-images.githubusercontent.com/1008395/106382331-31c79980-63bf-11eb-8312-e3c464f8ed75.mp4

**Smartphone (please complete the following information):**

_Reported on several phones. The following is just one example of a device where it did appear:_
- Device: OnePlus 5
- OS: Android 10
- App version and store: 1.0.14 - F-Droid