element-hq / element-hq/synapse
High postgresql CPU usage: `SELECT DISTINCT origin_chain_id FROM event_auth_chain_links WHERE origin_chain_id = (large list)`
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
### Description
We constantly have between 1 and 12 postgres processes using 100% CPU time. When I look at them, they're doing a query like this:
``` WITH RECURSIVE links(chain_id) AS (
SELECT
DISTINCT origin_chain_id
FROM event_auth_chain_links WHERE origin_chain_id = ANY(ARRAY[581501,1105790,581503,57216,581496,57218,57210,57211,57212,1105789,57214,57224,57225,2154370,57219,57221
,57229,57222,57223,57226,57227,2678668,581515,57236,57230,57238,57232,57240,57233,2154386,57234,57244,57237,57246,581534,581535,581537,57241,57251,57243,581532,57245,57255,581543,581545,5725
8,57259,57260,57252,57262,57254,581544,57265,57257,57267,57263,581550,57264,57266,57272,57268,57269,57275,581563,581564,57278,57279,57280,57281,57282,581566,57284,581569,581570,57287,581572,
57289,581573,581577,581578,581581,57285,581574,581575,57297,57298,57299,57300,57301,57302,57303,581585,581593,57306,581594,57308,57309,57310,57311,581598,581601,581596,581603,57316,581605,57
318,57319,581608,581609,57322,57315,57317,57325,57321,581606,57328,57329,581611,57331,1105908,57326,57327,57330,57332,1105910,57338,57335,2154487,57334,57342,57336,57337,2154491,57346,57341,
57343,57349,57344,57345,57347,57348,57350,57351,581644,57352,57353,57354,57355,57356,57357,57358,57364,57365,57359,57360,57361,57362,2154515,57371,57366,57373,57367,57368,2154521,57370,57372
,57374,57375,57376,57382,57383,57377,57378,57379,57380,57381,1105957,2154535,57384,57385,57393,581682,57387,581677,57389,57391,57392,57394,57395,57396,57403,57397,57398,57399,57400,57401,574
02,57404,57405,57406,57407,57408,57415,57409,57410,57411,57413,57414,57412,57416,57417,57424,57418,581714,57419,57420,57421,57422,57423,57425,1106009,57426,57427,57428,57437,581726,57439,574
31,57433,57434,57443,57435,57445,57446,57438,57448,57449,581730,57451,57452,57453,2678885,57455,57447,581736,57450,1106027,581748,57454,2678895,57456,2678897,57465,57466,57458,57459,57461,57
463,57464,581751,57467,57468,57469,57470,1106047,57478,57472,57473,57475,1106052,57483,57476,57477,57486,57487,57479,57489,57490,57482,57485,57484,581775,1106064,581784,57497,57498,57499,581
788,57501,57502,581790,57504,57496,581794,57507,581795,581796,57510,57503,57512,57505,57514,57506,57508,581797,57518,581806,57511,57521,57513,57523,57515,57516,57517,581807,57528,57520,58180
9,2678962,57525,57527,57526,57529,57530,57531,57532,57533,57540,1106110,57542,57535,57536,57537,57538,57539,57541,57549,57543,57544,57552,57545,57546,57547,57548,57550,57558,57551,57560,5755
3,57554,57555,57556,57565,57557,581847,57561,57562,57570,57564,57563,57573,57574,57566,57567,57568,57569,57571,57580,57572,57576,57577,57584,57578,57579,57581,57582,57583,57585,581873,57586,
57588,57594,57595,57589,581885,2154741,57590,57591,57593,57596,57603,57597,57598,57606,57600,57601,581897,57602,57611,1106187,57613,57605,57615,57616,57617,581904,57610,57620,57621,57622,581
911,57624,57618,57619,581915,1106197,581910,57630,1106207,57632,57633,581918,57635,57627,581925,57631,57634,581921,581929,57636,581924,57644,57645,1106222,57647,57648,57641,57642,57651,57646
,57653,57649,57650,57652,57654,57655,57657,57658,57661,57662,57659,57660,57656,57666,57667,1106244,581950,581958,57663,1106240,57673,57674,1106250,57676,57677,581964,57679,1106247,1106249,57
678,57680,581972,57683,57686,57687,57684,57681,581971,57682,57685,57688,57689,57690,57696,57697,57691,57693,57700,57694,57695,57698,57699,581987,57701,57707,581990,57702,57704,57705,57712,57
706,57708,57709,57716,57711,57710,57713,57714,57721,57715,57717,57718,57719,582008,57720,1106304,57729,57722,57723,57732,57724,57725,57726,57727,57728,582017,1106306,57731,57733,57734,57735,
57736,57737,57738,57747,57739,57741,57742,57751,57752,1106328,57745,57755,57756,57757,2679189,57759,2154903,57753,57754,1106331,57758,1106334,57766,57767,57760,57761,57770,57762,57763,57764,
57765,57775,57768,1106345,57769,57771,2679212,57773,57774,57776,57777,57778,2154931,57779,57780,57789,57781,57783,57784,57793,57786,57787,57788,1106373,57790,57791,57792,57794,57802,57795,57
796,57797,57798,57799,57808,2679240,57810,57803,57804,57805,57806,57807,1106385,57809,57811,57812,1106387,57813,57815,57816,57824,57825,57817,57818,1106404,57820,2154973,1106398,57832,57826,
57827,57828,1106403,57831,57830,57839,57833,57841,57834,57835,57836,57845,57838,57840,57842,57843,1106421,57846,57847,57844,57849,57848,57856,57851,57852,1106435,57860,57853,57854,57863,5786
4,57857,1106434,57859,57868,57869,1106444,57871,2679304,57873,57865,57866,57867,2679308,57878,57870,2155023,1106448,57874,57883,57875,57876,57877,57887,57888,57889,57890,57882,57884,57885,11
06463,57886,2155041,57891,57898,57899,57892,57893,57894,57895,57896,57905,57897,57900,57902,57909,57903,57911,57906,1106483,57908,57907,57910,57917,1106487,57912,57913,57914,57922,2679354,57
915,57916,57926,57927,57919,57920,57921,57923,2155076,57925,57928,2679369,57931,57937,57932,57933,57940,57934,57935,57936,57938,57939,57941,57942,2679382,1106520,2679385,57944,57947,57946,57950,57951,57952,57953,57954,57955,57956,57958,57959,1106539,57960,57957,57966,57962,57963,57964,57965,2155118,57967,57969,57970,57972,57973,57975,57976,57977,57980,57978,2679418,57983,57979,57985,57981,57987,57982,2679425,57986,57991,57984,57988,57989,57990,1106567,1106573,57992,57993,57994,57995,2679436,57996,58004,57998,58006,57999,58000,58001,58010,58002,58003,58005,58007,58008,58016,58009,1106587,58012,58020,58021,58013,58014,58015,58017,58018,58027,58028,58023,58024,58022,58026,2155176,58025,58029,58030,58031,58032,58033,58034,58035,58036,58037,58038,58039,58040,58047,58041,58042,58043,1106620,1106628,58044,58046,1106623,58048,58049,58058,58050,58060,58052,58053,58054,58055,58056,58057,58067,58061,58063,58062,58065,58072,58073,58066,58068,58069,58070,58071,58074,58076,58077,58079,58080,58078,58085,58083,58081,58082,58086,58087,58088,58089,58093,58090,58095,2679530,58097,58091,58092,58094,58096,58098,58099,58100,58101,58102,58103,58104,58105,58106,58107,58112,58108,58109,58110,58111,58113,58115,58114,58116,58118,58119,58123,58124,58120,58121,58122,58125,58126,1106706,58127,58128,58129,58130,58131,58136,58132,58133,58134,58135,58137,58138,58139,58140,58141,58142,58143,58148,1106725,58144,58145,58146,58147,58149,58150,58151,58152,58153,58159,58155,58156,58157,58158,58164,58160,58166,1106743,58161,58162,58163,58165,58167,58168,2679608,58169,58171,58172,58178,58173,58180,58175,58182,1106759,58184,58176,58177,2679627,58179,1106765,58189,58183,58185,58186,58194,58187,58196,58188,58190,58191,1106776,58193,58195,1106779,1106780,58203,58197,58207,58199,58209,1106777,58202,58204])
UNION
SELECT
target_chain_id
FROM event_auth_chain_links
INNER JOIN links ON (chain_id = origin_chain_id)
)
SELECT
origin_chain_id, origin_sequence_number,
target_chain_id, target_sequence_number
FROM links
INNER JOIN event_auth_chain_links ON (chain_id = origin_chain_id)
```
As far as I can tell, those queries actually only run for a few seconds, it just seems to be doing a lot of those.
We're running 1.137.0+bookworm1. We had this with older versions too, at least with 1.333.0.
I've manually ran an analyze, that didn't have any effect.
### Steps to reproduce
Run synapse
### Homeserver
matrix.debian.social
### Synapse Version
1.137.0+bookworm1
### Installation Method
Debian packages from packages.matrix.org
### Database
Running postgresql 15.13-0+deb12u1, no seperate server
### Workers
Single process
### Platform
Debian 12, in a container.
### Configuration
_No response_
### Relevant log output
```shell
Not sure what log output to provide
```
### Anything else that would be useful to know?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.