Custom graceful stop hooks (to support a truly "graceful" termination of runners)

What would you like added?

Allow ARC users to define custom "graceful stop hooks" (similar to "job started hooks" and "job completed hooks"),
that get executed during graceful_stop().

Why is this needed?

Github actions syntax allows defining "always() steps", like the following:

jobs:
  test:
    runs-on: self-hosted
    steps:
      - run: sleep 300s
      - run: echo 'this step always runs, even when workflow job is cancelled'
        if: always()

The end goal would be for self-hosted runners to also run these "always() steps" during their graceful_stop().

This would make the termination of self-hosted runners truly "graceful", since then users can define an "always() step" which (e.g.) sends an alert to slack notifying that a workflow job was cancelled prematurely and needs to be manually restarted if necessary.

To accomplish this end goal, we could either

1. Send a POST https://api.github.com/repos/{GITHUB_REPOSITORY}/actions/runs/{GITHUB_RUN_ID}/cancel request instead of doing kill -TERM "$runner_listener_pid" in graceful_stop().

or

2. Let ARC users(operators) accomplish the above themselves through custom "graceful stop hooks".

I'd think Option 1. would be a nice feature for ARC in general, but if it is preferred to take gradual steps, I'll be happy with at least just being able to define my own custom "graceful stop hooks" to be executed during graceful_stop().

Additional context

Add any other context or screenshots about the feature request here.

Start by reading runner/graceful-stop.sh, especially the existing kill command, and compare it with the referenced runner/hooks/job-started.sh and runner/hooks/job-completed.sh files. Determine how a configurable graceful-stop hook should be defined and invoked, then verify that it runs during graceful_stop() without breaking runner termination.