Bookshelf database cleanup tasks will never complete if they fall behind

Chef Infra Server 13.2 + Chef Backend 2.1, RHEL 7.8

bookshelf has two database cleanup tasks, do_delete_cleanup + do_upload_cleanup: https://github.com/chef/chef-server/blob/ac11df3d7c88b164a16beccdd18c12d403e87967/src/bookshelf/src/bksw_cleanup_task.erl#L79-L97

these tasks run every 7 and 19 minutes, respectively: https://github.com/chef/chef-server/blob/ac11df3d7c88b164a16beccdd18c12d403e87967/src/bookshelf/src/bksw_cleanup_task.erl#L52-L53

sqerl defaults to a 5 second postgres statement_timeout: https://github.com/chef/sqerl/blob/master/src/sqerl_client.erl#L68

If they take more than 5 seconds, bookshelf crashes:

2020-08-06 00:00:10 =SUPERVISOR REPORT====
     Supervisor: {local,bksw_sup}
     Context:    child_terminated
     Reason:     {timeout,{gen_server,call,[bksw_cleanup_task,delete_cleanup]}}
     Offender:   [{pid,<0.5948.224>},{id,bksw_cleanup_task},{mfargs,{bksw_cleanup_task,start_link,[]}},{restart_type,permanent},{shutdown,brutal_kill},{child_type,worker}]

This will continue forever until the cleanup is done manually, e.g.

echo "select from purge_expired(420000)\;" | su -l chef_pgsql -c 'psql bookshelf'

Other than the database size growing over time, we aren't sure of the side-affects of this behavior at this time.

Aha! Link: https://chef.aha.io/features/SH-3062

Start with src/bookshelf/src/bksw_cleanup_task.erl at lines 52-53 and 79-97, then inspect sqerl_client.erl at line 68 to understand the timeout interaction. Review the linked Aha feature SH-3062 for the intended resolution; the issue does not define a standalone completion condition.