implement workspaces supervisor

Currently, whenever the agent crash or is killed, the workspace has to be restarted in order for the agent to also be started again.

As part of a bigger effort to improve reliability of the agents - we investigated having a supervisor started on the workspaces that will monitor the agent and ensure it is running.

### Current situation

Currently, whenever a workspace is started, a script (based on the arch) [from there](https://github.com/coder/coder/tree/36c2cf8a408111fc2a8ac6f5746393ea818e0524/provisionersdk/scripts) is executed. This one will download the corresponding agent and start it.

Going one step before and looking at the `docker_container` resource inside `main.tf` from a template - we can see the following line :

```
entrypoint = ["sh", "-c", coder_agent.dev.init_script]
```

This entrypoint will execute the script described above.

### Investigations

What we tried to do is to open a terminal into a running workspace - and run `supervisord`.
With supervisord running, we've been able to run and monitor an external script.

Script :

```sh
# Define the file to watch
FILE_TO_WATCH="/path/to/your/file"

echo "Waiting for file: $FILE_TO_WATCH"

# Loop until the file is found
while true; do
if [ -f "$FILE_TO_WATCH" ]; then
echo "File found: $FILE_TO_WATCH"
exit 0
fi
# Sleep for a short duration to avoid busy-waiting
sleep 1
done
```

Configuration used to watch the file :
```
[program:idle.sh]
command=/home/coder/idle.sh
autostart=true
autorestart=true
stdout_logfile=/dev/fd/1
stderr_logfile=/dev/fd/2
stdout_logfile_maxbytes=0
stderr_logfile_maxbytes=0
```

Then, if we create the file matching FILE_TO_WATCH - we can see that the process is killed, spawned (by supervisord) and killed infinitely - until we delete the file, and so the file is back to normal.

```
Waiting for file
Waiting for file
File found: /home/coder/file_to_watch
2024-12-12 11:49:01,244 INFO exited: idle.sh (exit status 0; expected)
2024-12-12 11:49:02,246 INFO spawned: 'idle.sh' with pid 73778
Waiting for file: /home/coder/file_to_watch
File found: /home/coder/file_to_watch
2024-12-12 11:49:02,252 WARN exited: idle.sh (exit status 0; not expected)
2024-12-12 11:49:03,254 INFO spawned: 'idle.sh' with pid 73781
Waiting for file: /home/coder/file_to_watch
File found: /home/coder/file_to_watch
2024-12-12 11:49:03,261 WARN exited: idle.sh (exit status 0; not expected)
2024-12-12 11:49:05,265 INFO spawned: 'idle.sh' with pid 73808
Waiting for file: /home/coder/file_to_watch
File found: /home/coder/file_to_watch
2024-12-12 11:49:05,271 WARN exited: idle.sh (exit status 0; not expected)
2024-12-12 11:49:08,276 INFO spawned: 'idle.sh' with pid 73876
Waiting for file: /home/coder/file_to_watch
Waiting for file
2024-12-12 11:49:09,283 INFO success: idle.sh entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Waiting for file
Waiting for file
```

### Results & Next steps

This PoC validates that we can have a script / binary running, and that supervisord will make sure it is always healthy.
Being able to do it in a reliable way would be a huge improvement for us and for the agent part.

Next steps to validate are :

- How to ensure we have a solution running does not matter the architecture and environment
- How to configure the supervisor monitoring to ensure an agent is considered as healthy only based on some conditions
- The child process is detached from the supervisor one - and so the supervisor (if custom one) is not killed if too much memory is used.

#### Proposals

__ Supervisord

A first option to investigate is `supervisord`.
The positive point is that this is a well known, maintained and reliable option that we know work.
The problem here is the compatibility as we have to find an equivalent solution for platforms not-compatible with it.

__Custom supervisor

Another option to investigate is to have a `custom supervisor` - being a binary injected in the workspace and running.
The positive point is that we have full control of the code here, can create a custom and open-source library and improve the compatibility with Go and staticaly-linked binaries.
The problem is that it is one more piece of code to maintain on our side - for a critical component.
ℹ It can be worth having a look at the github repositories or libraries that exist and can potentially be used or wrapped.