Default secondary sort may cause performance issues

Someone discovered that a workaround for this slowdown was including a two column index:

alter table ccdb.app_usage_events add index ix_workaround (created_at desc, guid asc);

When app_usage_events has 5.7 million rows:

Before index:

SELECT count(*) AS `count` FROM `app_usage_events` LIMIT 1;
1 row in set (1.32 sec)

SELECT * FROM `app_usage_events` ORDER BY `app_usage_events`.`created_at` DESC, `app_usage_events`.`guid` ASC LIMIT 1 OFFSET 0;
1 row in set (13.58 sec)

After index:

SELECT count(*) AS `count` FROM `app_usage_events` LIMIT 1;
1 row in set (3.89 sec)

SELECT * FROM `app_usage_events` ORDER BY `app_usage_events`.`created_at` DESC, `app_usage_events`.`guid` ASC LIMIT 1 OFFSET 0;
1 row in set (0.00 sec)

We haven't personally done any testing on this performance improvement but it suggests we may want to revisit this line:

https://github.com/cloudfoundry/cloud_controller_ng/blob/main/lib/cloud_controller/paging/sequel_paginator.rb#L16

in which we automatically add a secondary sort to all queries to preserve a consistent order when returning results.

Here is the original commit: https://github.com/cloudfoundry/cloud_controller_ng/commit/fedda54b8471cf137e0ee15e7ba5de7350aa8f25

We may not have considered the performance implications of always including a secondary sort on GUID without adding a two column index for each sortable field in each table. Perhaps we want to find another way to guarantee consistent results order, or only add this index for created_at and other common sorts.

Start with lib/cloud_controller/paging/sequel_paginator.rb#L16 and the original commit linked in the issue to understand why the secondary GUID sort is added. Reproduce the reported ORDER BY timings on app_usage_events with and without the two-column index, then determine and document a tested approach for preserving consistent ordering without unacceptable query performance.