Response time on "cf apps" call on v2 vs. v3 api

Issue

We measured the response time on cf apps calls of the v2 api and the v3 api and observed a significantly increased response time on the v3 api.

Context

On one of our large landscapes we have a space with ~700 apps.
If cf apps gets called it takes about 4-5 minutes which is barely acceptable for customers. However if we use cf7 apps the response time increases to ~2 hours (Yes we triple checked this) which is absolutely not acceptable. We dug deeper into this and found the following:

In the V2 api it is just called:

• /v2/spaces/someguid/summary - This call alone takes 300-450 seconds to come back.
  In the space with 700 apps it made about 111000 select statements during this time period.
  So the time increase per app on that scale seems to be ~0.5 - 1 second.

In the V3 api there are several calls made:
First to get all the apps:

• /v3/apps?space_guids=someguid - This call takes about 5 seconds (totally fine)

The Problem seems to be that after the initial call that gets all apps, for every app found there are more calls:

• /v3/apps/someguid/routes - This call takes about 4-6 seconds
• /v3/apps/someguid/processes - This call takes about 1-2 second
• /v3/processes/someguid/sidecars - This call takes about 1-2 second
• /v3/processes/someguid/stats - This call takes about 1-2 second
  ... that are the ones that took the longest.

That adds up to a total time of ~700 apps x ~10 seconds = ~7000 seconds = ~2 hours.

So the time increase per app on that scale seems to be 10 seconds.

Steps to Reproduce

Not sure if reproduce-able due to the size of the landscape this problem was observed on. We were not able to reproduce the exact times mentioned here on a development landscape. Nevertheless we observed, that the time increase and the number of calls that are executed for cf apps and cf7 apps scale with the number of apps per space.

Expected result

300-450 seconds is barely acceptable for customers working with the cli and websites which use the cli to display all apps in a space are not usable at all. But the v2 (cf apps) can still be used with a bit of patience. However the v3 api (cf7 apps) with more than 2 hours of loading time is not useable with the mentioned amount of apps in a space. We would expect the loading time of the v3 api to be at least as fast as the v2 api.

Current result

The v2 api outperforms the v3 api on a big landscape that has a space with ~700 apps in it on the cf apps (cf7 apps) call.

Start by profiling the cf apps and cf7 apps entry points, comparing /v2/spaces/:guid/summary with /v3/apps?space_guids=:guid and the per-app routes, processes, sidecars, and stats calls. Use a large space to measure query counts and latency; done means the v3 path no longer scales into hours for roughly 700 apps and is at least comparable to v2.