[Argus] `heap out of memory` crash

Full log with crash at the end:
```
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101959
{
"meta": {
"req": {
"url": "/api/v1/assets/2101959",
"headers": {
"host": "dist1.joyutils.org",
"user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
"accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
"accept-encoding": "gzip",
"accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
"cdn-loop": "cloudflare",
"cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
"cf-ipcountry": "AR",
"cf-ray": "8590cd7e2922ab32-EZE",
"cf-visitor": "{\"scheme\":\"https\"}",
"referer": "https://gleev.xyz/",
"sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
"sec-ch-ua-mobile": "?1",
"sec-ch-ua-platform": "\"Android\"",
"sec-fetch-dest": "image",
"sec-fetch-mode": "no-cors",
"sec-fetch-site": "cross-site",
"traceparent": "00-5ccdb8e7a7da8b8a17c9639cdfa34827-d6262f86b925a15e-01",
"x-forwarded-for": "172.69.35.139",
"x-forwarded-host": "dist1.joyutils.org",
"x-forwarded-proto": "https"
},
"method": "GET",
"httpVersion": "1.1",
"originalUrl": "/api/v1/assets/2101959",
"query": {}
},
"res": {
"headers": {
"x-powered-by": "Express",
"access-control-allow-origin": "*",
"timing-allow-origin": "*",
"access-control-expose-headers": "x-cache, x-data-source",
"content-type": "application/json; charset=utf-8",
"content-length": "105",
"etag": "W/\"69-YoCNAtC7v+WLZhCrBKuH2VsJUis\""
},
"statusCode": 500
},
"responseTime": 27719,
"prematurelyClosed": false
},
"trace_id": "5ccdb8e7a7da8b8a17c9639cdfa34827",
"span_id": "b9b56f20f279f540",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://andybut.xyz/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
"span_id": "5f669cb66119a1e9",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://andybut.xyz/storage/api/v1"
},
"objectId": "2101915",
"err": {
"message": "Request timeout"
},
"trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
"span_id": "5f669cb66119a1e9",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://stor1.joyutils.org/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
"span_id": "5f669cb66119a1e9",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://stor1.joyutils.org/storage/api/v1"
},
"objectId": "2101915",
"err": {
"message": "Request timeout"
},
"trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
"span_id": "5f669cb66119a1e9",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
"span_id": "5f669cb66119a1e9",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
},
"objectId": "2101915",
"err": {
"message": "Request timeout"
},
"trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
"span_id": "5f669cb66119a1e9",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
"span_id": "5f669cb66119a1e9",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
},
"objectId": "2101915",
"err": {
"message": "Request timeout"
},
"trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
"span_id": "5f669cb66119a1e9",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 NetworkingManager error: Data object download failed
{
"err": {
"message": "Failed to download object 2101915 from any availablable storage provider",
"stack": "Error: Failed to download object 2101915 from any availablable storage provider\n at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:219:24)\n at Queue. (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n at Queue.emit (node:events:517:28)\n at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n at next (/joystream/node_modules/queue/index.js:118:16)\n at /joystream/node_modules/queue/index.js:150:14\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at runNextTicks (node:internal/process/task_queues:64:3)\n at listOnTimeout (node:internal/timers:538:9)\n at process.processTimers (node:internal/timers:512:7)"
},
"trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
"span_id": "5f669cb66119a1e9",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi error: middlewareError
{
"err": {
"message": "Failed to download object 2101915 from any availablable storage provider",
"stack": "Error: Failed to download object 2101915 from any availablable storage provider\n at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:218:25)\n at Queue. (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n at Queue.emit (node:events:517:28)\n at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n at next (/joystream/node_modules/queue/index.js:118:16)\n at /joystream/node_modules/queue/index.js:150:14\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at runNextTicks (node:internal/process/task_queues:64:3)\n at listOnTimeout (node:internal/timers:538:9)\n at process.processTimers (node:internal/timers:512:7)"
},
"req": {
"url": "/api/v1/assets/2101915",
"headers": {
"host": "dist1.joyutils.org",
"user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
"accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
"accept-encoding": "gzip",
"accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
"cdn-loop": "cloudflare",
"cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
"cf-ipcountry": "AR",
"cf-ray": "8590cd7e2925ab32-EZE",
"cf-visitor": "{\"scheme\":\"https\"}",
"referer": "https://gleev.xyz/",
"sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
"sec-ch-ua-mobile": "?1",
"sec-ch-ua-platform": "\"Android\"",
"sec-fetch-dest": "image",
"sec-fetch-mode": "no-cors",
"sec-fetch-site": "cross-site",
"traceparent": "00-a93216e4db2d4bb4c93708c9b31c5b71-0bd49bf08deb91aa-01",
"x-forwarded-for": "172.69.35.138",
"x-forwarded-host": "dist1.joyutils.org",
"x-forwarded-proto": "https"
},
"method": "GET",
"httpVersion": "1.1",
"originalUrl": "/api/v1/assets/2101915",
"query": {}
},
"trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
"span_id": "5f669cb66119a1e9",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101915
{
"meta": {
"req": {
"url": "/api/v1/assets/2101915",
"headers": {
"host": "dist1.joyutils.org",
"user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
"accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
"accept-encoding": "gzip",
"accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
"cdn-loop": "cloudflare",
"cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
"cf-ipcountry": "AR",
"cf-ray": "8590cd7e2925ab32-EZE",
"cf-visitor": "{\"scheme\":\"https\"}",
"referer": "https://gleev.xyz/",
"sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
"sec-ch-ua-mobile": "?1",
"sec-ch-ua-platform": "\"Android\"",
"sec-fetch-dest": "image",
"sec-fetch-mode": "no-cors",
"sec-fetch-site": "cross-site",
"traceparent": "00-a93216e4db2d4bb4c93708c9b31c5b71-0bd49bf08deb91aa-01",
"x-forwarded-for": "172.69.35.138",
"x-forwarded-host": "dist1.joyutils.org",
"x-forwarded-proto": "https"
},
"method": "GET",
"httpVersion": "1.1",
"originalUrl": "/api/v1/assets/2101915",
"query": {}
},
"res": {
"headers": {
"x-powered-by": "Express",
"access-control-allow-origin": "*",
"timing-allow-origin": "*",
"access-control-expose-headers": "x-cache, x-data-source",
"content-type": "application/json; charset=utf-8",
"content-length": "105",
"etag": "W/\"69-17VO0ULVLc30wSWRZevqEyq7CAk\""
},
"statusCode": 500
},
"responseTime": 27720,
"prematurelyClosed": false
},
"trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
"span_id": "5f669cb66119a1e9",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://andybut.xyz/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "d1fcecc237c7860294550a8740333cf0",
"span_id": "0b96bb65ac3d7e19",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://andybut.xyz/storage/api/v1"
},
"objectId": "2101979",
"err": {
"message": "Request timeout"
},
"trace_id": "d1fcecc237c7860294550a8740333cf0",
"span_id": "0b96bb65ac3d7e19",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://stor1.joyutils.org/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "d1fcecc237c7860294550a8740333cf0",
"span_id": "0b96bb65ac3d7e19",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://stor1.joyutils.org/storage/api/v1"
},
"objectId": "2101979",
"err": {
"message": "Request timeout"
},
"trace_id": "d1fcecc237c7860294550a8740333cf0",
"span_id": "0b96bb65ac3d7e19",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "d1fcecc237c7860294550a8740333cf0",
"span_id": "0b96bb65ac3d7e19",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
},
"objectId": "2101979",
"err": {
"message": "Request timeout"
},
"trace_id": "d1fcecc237c7860294550a8740333cf0",
"span_id": "0b96bb65ac3d7e19",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "d1fcecc237c7860294550a8740333cf0",
"span_id": "0b96bb65ac3d7e19",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
},
"objectId": "2101979",
"err": {
"message": "Request timeout"
},
"trace_id": "d1fcecc237c7860294550a8740333cf0",
"span_id": "0b96bb65ac3d7e19",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 NetworkingManager error: Data object download failed
{
"err": {
"message": "Failed to download object 2101979 from any availablable storage provider",
"stack": "Error: Failed to download object 2101979 from any availablable storage provider\n at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:219:24)\n at Queue. (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n at Queue.emit (node:events:517:28)\n at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n at next (/joystream/node_modules/queue/index.js:118:16)\n at /joystream/node_modules/queue/index.js:150:14\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at runNextTicks (node:internal/process/task_queues:64:3)\n at listOnTimeout (node:internal/timers:538:9)\n at process.processTimers (node:internal/timers:512:7)"
},
"trace_id": "d1fcecc237c7860294550a8740333cf0",
"span_id": "0b96bb65ac3d7e19",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi error: middlewareError
{
"err": {
"message": "Failed to download object 2101979 from any availablable storage provider",
"stack": "Error: Failed to download object 2101979 from any availablable storage provider\n at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:218:25)\n at Queue. (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n at Queue.emit (node:events:517:28)\n at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n at next (/joystream/node_modules/queue/index.js:118:16)\n at /joystream/node_modules/queue/index.js:150:14\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at runNextTicks (node:internal/process/task_queues:64:3)\n at listOnTimeout (node:internal/timers:538:9)\n at process.processTimers (node:internal/timers:512:7)"
},
"req": {
"url": "/api/v1/assets/2101979",
"headers": {
"host": "dist1.joyutils.org",
"user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
"accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
"accept-encoding": "gzip",
"accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
"cdn-loop": "cloudflare",
"cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
"cf-ipcountry": "AR",
"cf-ray": "8590cd7e2912ab32-EZE",
"cf-visitor": "{\"scheme\":\"https\"}",
"referer": "https://gleev.xyz/",
"sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
"sec-ch-ua-mobile": "?1",
"sec-ch-ua-platform": "\"Android\"",
"sec-fetch-dest": "image",
"sec-fetch-mode": "no-cors",
"sec-fetch-site": "cross-site",
"traceparent": "00-d1fcecc237c7860294550a8740333cf0-91e764aa10290763-01",
"x-forwarded-for": "172.69.35.139",
"x-forwarded-host": "dist1.joyutils.org",
"x-forwarded-proto": "https"
},
"method": "GET",
"httpVersion": "1.1",
"originalUrl": "/api/v1/assets/2101979",
"query": {}
},
"trace_id": "d1fcecc237c7860294550a8740333cf0",
"span_id": "0b96bb65ac3d7e19",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101979
{
"meta": {
"req": {
"url": "/api/v1/assets/2101979",
"headers": {
"host": "dist1.joyutils.org",
"user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
"accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
"accept-encoding": "gzip",
"accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
"cdn-loop": "cloudflare",
"cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
"cf-ipcountry": "AR",
"cf-ray": "8590cd7e2912ab32-EZE",
"cf-visitor": "{\"scheme\":\"https\"}",
"referer": "https://gleev.xyz/",
"sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
"sec-ch-ua-mobile": "?1",
"sec-ch-ua-platform": "\"Android\"",
"sec-fetch-dest": "image",
"sec-fetch-mode": "no-cors",
"sec-fetch-site": "cross-site",
"traceparent": "00-d1fcecc237c7860294550a8740333cf0-91e764aa10290763-01",
"x-forwarded-for": "172.69.35.139",
"x-forwarded-host": "dist1.joyutils.org",
"x-forwarded-proto": "https"
},
"method": "GET",
"httpVersion": "1.1",
"originalUrl": "/api/v1/assets/2101979",
"query": {}
},
"res": {
"headers": {
"x-powered-by": "Express",
"access-control-allow-origin": "*",
"timing-allow-origin": "*",
"access-control-expose-headers": "x-cache, x-data-source",
"content-type": "application/json; charset=utf-8",
"content-length": "105",
"etag": "W/\"69-GuyLFrfFSeAf+cgtCn08QrrAopo\""
},
"statusCode": 500
},
"responseTime": 27726,
"prematurelyClosed": false
},
"trace_id": "d1fcecc237c7860294550a8740333cf0",
"span_id": "0b96bb65ac3d7e19",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://andybut.xyz/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "967d1636c501a5f625c64943e74227dc",
"span_id": "1c64976e266f47d3",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://andybut.xyz/storage/api/v1"
},
"objectId": "2101965",
"err": {
"message": "Request timeout"
},
"trace_id": "967d1636c501a5f625c64943e74227dc",
"span_id": "1c64976e266f47d3",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://stor1.joyutils.org/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "967d1636c501a5f625c64943e74227dc",
"span_id": "1c64976e266f47d3",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://stor1.joyutils.org/storage/api/v1"
},
"objectId": "2101965",
"err": {
"message": "Request timeout"
},
"trace_id": "967d1636c501a5f625c64943e74227dc",
"span_id": "1c64976e266f47d3",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "967d1636c501a5f625c64943e74227dc",
"span_id": "1c64976e266f47d3",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
},
"objectId": "2101965",
"err": {
"message": "Request timeout"
},
"trace_id": "967d1636c501a5f625c64943e74227dc",
"span_id": "1c64976e266f47d3",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "967d1636c501a5f625c64943e74227dc",
"span_id": "1c64976e266f47d3",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
},
"objectId": "2101965",
"err": {
"message": "Request timeout"
},
"trace_id": "967d1636c501a5f625c64943e74227dc",
"span_id": "1c64976e266f47d3",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 NetworkingManager error: Data object download failed
{
"err": {
"message": "Failed to download object 2101965 from any availablable storage provider",
"stack": "Error: Failed to download object 2101965 from any availablable storage provider\n at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:219:24)\n at Queue. (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n at Queue.emit (node:events:517:28)\n at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n at next (/joystream/node_modules/queue/index.js:118:16)\n at /joystream/node_modules/queue/index.js:150:14\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at runNextTicks (node:internal/process/task_queues:64:3)\n at listOnTimeout (node:internal/timers:538:9)\n at process.processTimers (node:internal/timers:512:7)"
},
"trace_id": "967d1636c501a5f625c64943e74227dc",
"span_id": "1c64976e266f47d3",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi error: middlewareError
{
"err": {
"message": "Failed to download object 2101965 from any availablable storage provider",
"stack": "Error: Failed to download object 2101965 from any availablable storage provider\n at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:218:25)\n at Queue. (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n at Queue.emit (node:events:517:28)\n at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n at next (/joystream/node_modules/queue/index.js:118:16)\n at /joystream/node_modules/queue/index.js:150:14\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at runNextTicks (node:internal/process/task_queues:64:3)\n at listOnTimeout (node:internal/timers:538:9)\n at process.processTimers (node:internal/timers:512:7)"
},
"req": {
"url": "/api/v1/assets/2101965",
"headers": {
"host": "dist1.joyutils.org",
"user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
"accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
"accept-encoding": "gzip",
"accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
"cdn-loop": "cloudflare",
"cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
"cf-ipcountry": "AR",
"cf-ray": "8590cd7e292dab32-EZE",
"cf-visitor": "{\"scheme\":\"https\"}",
"referer": "https://gleev.xyz/",
"sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
"sec-ch-ua-mobile": "?1",
"sec-ch-ua-platform": "\"Android\"",
"sec-fetch-dest": "image",
"sec-fetch-mode": "no-cors",
"sec-fetch-site": "cross-site",
"traceparent": "00-967d1636c501a5f625c64943e74227dc-74de9b66e6252cb1-01",
"x-forwarded-for": "172.69.35.139",
"x-forwarded-host": "dist1.joyutils.org",
"x-forwarded-proto": "https"
},
"method": "GET",
"httpVersion": "1.1",
"originalUrl": "/api/v1/assets/2101965",
"query": {}
},
"trace_id": "967d1636c501a5f625c64943e74227dc",
"span_id": "1c64976e266f47d3",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101965
{
"meta": {
"req": {
"url": "/api/v1/assets/2101965",
"headers": {
"host": "dist1.joyutils.org",
"user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
"accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
"accept-encoding": "gzip",
"accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
"cdn-loop": "cloudflare",
"cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
"cf-ipcountry": "AR",
"cf-ray": "8590cd7e292dab32-EZE",
"cf-visitor": "{\"scheme\":\"https\"}",
"referer": "https://gleev.xyz/",
"sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
"sec-ch-ua-mobile": "?1",
"sec-ch-ua-platform": "\"Android\"",
"sec-fetch-dest": "image",
"sec-fetch-mode": "no-cors",
"sec-fetch-site": "cross-site",
"traceparent": "00-967d1636c501a5f625c64943e74227dc-74de9b66e6252cb1-01",
"x-forwarded-for": "172.69.35.139",
"x-forwarded-host": "dist1.joyutils.org",
"x-forwarded-proto": "https"
},
"method": "GET",
"httpVersion": "1.1",
"originalUrl": "/api/v1/assets/2101965",
"query": {}
},
"res": {
"headers": {
"x-powered-by": "Express",
"access-control-allow-origin": "*",
"timing-allow-origin": "*",
"access-control-expose-headers": "x-cache, x-data-source",
"content-type": "application/json; charset=utf-8",
"content-length": "105",
"etag": "W/\"69-e0TQIzyfoufzzJX6e7Fyy/VdXdA\""
},
"statusCode": 500
},
"responseTime": 27739,
"prematurelyClosed": false
},
"trace_id": "967d1636c501a5f625c64943e74227dc",
"span_id": "1c64976e266f47d3",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://andybut.xyz/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "06efab4bfed59b61dade00ff424919e7",
"span_id": "76c73c02df4913f4",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://andybut.xyz/storage/api/v1"
},
"objectId": "2101963",
"err": {
"message": "Request timeout"
},
"trace_id": "06efab4bfed59b61dade00ff424919e7",
"span_id": "76c73c02df4913f4",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://stor1.joyutils.org/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "06efab4bfed59b61dade00ff424919e7",
"span_id": "76c73c02df4913f4",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://stor1.joyutils.org/storage/api/v1"
},
"objectId": "2101963",
"err": {
"message": "Request timeout"
},
"trace_id": "06efab4bfed59b61dade00ff424919e7",
"span_id": "76c73c02df4913f4",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "06efab4bfed59b61dade00ff424919e7",
"span_id": "76c73c02df4913f4",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
},
"objectId": "2101963",
"err": {
"message": "Request timeout"
},
"trace_id": "06efab4bfed59b61dade00ff424919e7",
"span_id": "76c73c02df4913f4",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
"0": {
"endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
},
"timeoutMs": 5000,
"trace_id": "06efab4bfed59b61dade00ff424919e7",
"span_id": "76c73c02df4913f4",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
"0": {
"endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
},
"objectId": "2101963",
"err": {
"message": "Request timeout"
},
"trace_id": "06efab4bfed59b61dade00ff424919e7",
"span_id": "76c73c02df4913f4",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 NetworkingManager error: Data object download failed
{
"err": {
"message": "Failed to download object 2101963 from any availablable storage provider",
"stack": "Error: Failed to download object 2101963 from any availablable storage provider\n at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:219:24)\n at Queue. (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n at Queue.emit (node:events:517:28)\n at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n at next (/joystream/node_modules/queue/index.js:118:16)\n at /joystream/node_modules/queue/index.js:150:14\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at runNextTicks (node:internal/process/task_queues:64:3)\n at listOnTimeout (node:internal/timers:538:9)\n at process.processTimers (node:internal/timers:512:7)"
},
"trace_id": "06efab4bfed59b61dade00ff424919e7",
"span_id": "76c73c02df4913f4",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi error: middlewareError
{
"err": {
"message": "Failed to download object 2101963 from any availablable storage provider",
"stack": "Error: Failed to download object 2101963 from any availablable storage provider\n at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:218:25)\n at Queue. (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n at Queue.emit (node:events:517:28)\n at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n at next (/joystream/node_modules/queue/index.js:118:16)\n at /joystream/node_modules/queue/index.js:150:14\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at runNextTicks (node:internal/process/task_queues:64:3)\n at listOnTimeout (node:internal/timers:538:9)\n at process.processTimers (node:internal/timers:512:7)"
},
"req": {
"url": "/api/v1/assets/2101963",
"headers": {
"host": "dist1.joyutils.org",
"user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
"accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
"accept-encoding": "gzip",
"accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
"cdn-loop": "cloudflare",
"cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
"cf-ipcountry": "AR",
"cf-ray": "8590cd7c5e85ab32-EZE",
"cf-visitor": "{\"scheme\":\"https\"}",
"referer": "https://gleev.xyz/",
"sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
"sec-ch-ua-mobile": "?1",
"sec-ch-ua-platform": "\"Android\"",
"sec-fetch-dest": "image",
"sec-fetch-mode": "no-cors",
"sec-fetch-site": "cross-site",
"traceparent": "00-06efab4bfed59b61dade00ff424919e7-b880249753ee0cfd-01",
"x-forwarded-for": "172.69.35.138",
"x-forwarded-host": "dist1.joyutils.org",
"x-forwarded-proto": "https"
},
"method": "GET",
"httpVersion": "1.1",
"originalUrl": "/api/v1/assets/2101963",
"query": {}
},
"trace_id": "06efab4bfed59b61dade00ff424919e7",
"span_id": "76c73c02df4913f4",
"trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101963
{
"meta": {
"req": {
"url": "/api/v1/assets/2101963",
"headers": {
"host": "dist1.joyutils.org",
"user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
"accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
"accept-encoding": "gzip",
"accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
"cdn-loop": "cloudflare",
"cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
"cf-ipcountry": "AR",
"cf-ray": "8590cd7c5e85ab32-EZE",
"cf-visitor": "{\"scheme\":\"https\"}",
"referer": "https://gleev.xyz/",
"sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
"sec-ch-ua-mobile": "?1",
"sec-ch-ua-platform": "\"Android\"",
"sec-fetch-dest": "image",
"sec-fetch-mode": "no-cors",
"sec-fetch-site": "cross-site",
"traceparent": "00-06efab4bfed59b61dade00ff424919e7-b880249753ee0cfd-01",
"x-forwarded-for": "172.69.35.138",
"x-forwarded-host": "dist1.joyutils.org",
"x-forwarded-proto": "https"
},
"method": "GET",
"httpVersion": "1.1",
"originalUrl": "/api/v1/assets/2101963",
"query": {}
},
"res": {
"headers": {
"x-powered-by": "Express",
"access-control-allow-origin": "*",
"timing-allow-origin": "*",
"access-control-expose-headers": "x-cache, x-data-source",
"content-type": "application/json; charset=utf-8",
"content-length": "105",
"etag": "W/\"69-tiUt7lt6aYy57dMAQTIHVELHQO4\""
},
"statusCode": 500
},
"responseTime": 27752,
"prematurelyClosed": false
},
"trace_id": "06efab4bfed59b61dade00ff424919e7",
"span_id": "76c73c02df4913f4",
"trace_flags": "01"
}

<--- Last few GCs --->

[7:0x55f3870] 106762850 ms: Scavenge 4039.2 (4114.6) -> 4038.9 (4125.6) MB, 8.6 / 0.0 ms (average mu = 0.253, current mu = 0.152) allocation failure;
[7:0x55f3870] 106762872 ms: Scavenge 4045.8 (4125.6) -> 4045.6 (4126.4) MB, 14.1 / 0.0 ms (average mu = 0.253, current mu = 0.152) allocation failure;
[7:0x55f3870] 106764336 ms: Scavenge 4046.6 (4126.4) -> 4045.8 (4148.4) MB, 1463.1 / 0.0 ms (average mu = 0.253, current mu = 0.152) allocation failure;

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
1: 0xb95b60 node::Abort() [node]
2: 0xa9a7f8 [node]
3: 0xd6f2f0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xd6f697 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xf4cba5 [node]
6: 0xf5f08d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
7: 0xf3978e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0xf3ab57 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
9: 0xf1bd2a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x12e114f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x170deb9 [node]
/joystream/distributor-node/runner.sh: line 8: 7 Aborted (core dumped) node --require @joystream/opentelemetry ./bin/run $*
Loaded Application Instrumentation: "Distributor Node"
Starting tracing...
2024-02-21 17:46:38:4638 App info: Starting the app
```
All of the above logs claim to be at the exact same timestamp which is a bit weird.


Looks like a memory leak, not sure what happened on 19th