Devcontainer Integration (coderd, agent, ui and IDEs)

Coder was the first alternative to `codespaces`.
But as of Oct. 2024 it lack behind `gitpod` and `daytona` in term of `devcontainer` support.
IMHO this is a critical issue that should be fixed.

### Context

Since the first release of `envbuilder` in sept 2023 we got a bunch of really nice releases notes. 🚀
But none included a mention of some form of devcontainer integration with `coder`. 💔

Coder's concurrents don't care about re-implementing the whole `devcontainer spec`.
Instead they just use the [devcontainer cli][devcontainer-cli] and focus their time on the `UX` and `integration`.

The only advantage of `envbuilder` I see is for running workspaces as pod, that's it.
For workspaces as VM `envbuilder` currently does not make any sense to me.
The [official implem][devcontainer-cli] is currently much better because it implements the whole spec.
(prebuilds, secrets, merge configs (features, extensions, etc..), output effective conf, lock file, dotfiles, compose...)

As a user I really like coder's flexibility, but the devcontainer DevX is just bad.
It's makes me reconsider less flexible alternatives like `daytona`.

I hope `coder` will focus now on filling the feature parity gap with `daytona`, `gitpod` and hopefully `codespaces`.

---

### Implementation Proposal

In a similar way github do, `coder` should query the SCMaaS provider to know which branches exists and fetch the current configuration.


In short `coderd` and the `agent` must be able to parse a devcontainer config.
Respectively before `provisioning time` and during `runtime`.

Before the workspace creation form is submitted `coderd` must fetch and parse the devcontainer config. (for features like [secrets][secret-implem], prebuilds, hardware requirements... )

#### How

1. Add an `SCM provider` interface + implement each supported providers ( github, gitlab, bitbucket server...)
2. Add a new field `kind` in `coder_parameter`. (when `kind` is set `validate` field become useless)
Each values bind to it's own method in `coderd`.
Theses methods will autocomplete / validate the `coder_parameter` through websockets. (they'll use the scm interface for fetching data)
Config example :
```yaml
data "coder_parameter" "repository_url" {
name = "repository_url"
display_name = "Repository URL"
default = "https://hostname/.../repo.git"
mutable = true
# The server would use this field to know it's the repository url.
kind = "git_repository"
# It could also guess the coder_external_auth with the hostname in the url.
# Not a great UX though, no drop down list of projects and repositories...
# Could use three separate fields instead, ex: scm_provider (gitlab, bitbucket...), scm_project and scm_repository
}
data "coder_parameter" "target_branch" {
name = "target_branch"
display_name = "Branch"
mutable = true
kind = "git_branch" # point to some server side code that will query the git provider
}
data "coder_parameter" "devcontainer_config" {
name = "devcontainer_config"
display_name = "Devcontainer configuration"
default = ".devcontainer/devcontainer.json" # ignored anyway, file may not exists
mutable = true
kind = "devcontainer_config" # coderd will complete / validate and fetch/parse the file right away for secrets and other features)
}
# we could add other values for validating `hostRequirements`... ram, cpu, storage, GPUs...
# eg: devcontainer_ram, devcontainer_cpu, devcontainer_storage...
```

First it will improve the DevX with completion / validation on fields like `branch` , `devcontainer config`.
And then enable implementing important features such as [devcontainer secrets][secret-implem] and `prebuilds`...

#### Going further

The proposal above is quite flexible and could be reused outside of `devcontainers`.

The SCM provider could be extended for other use-cases.
As example it could be reused for features related to gitops and the new [coderd tf provider](https://github.com/coder/terraform-provider-coderd).
It could show infos of the gitops repo(s) right in the UI or CLI. (taking inspiration from `argocd` )
It also enable fetching a config file like `coder.yaml` from a repository to configure features like prebuilds.

The `kind` field can also be used for other purposes.
We could think of a way to let template admins define their own custom `values` and `methods` to validate `coder_parameters`.
Some examples would be validate / autocomplete :
- An artifact path from `artifactory` or equivalent.
- Selecting data from k8s or any cloud provider api
- An instance_type or regions
- Selecting a deployed resource name.
- Validating that a resource name is available or not.
- ?

Any feedback / enhancements welcomed, feel free to reach out to me on slack/discord
Happy to help

---

If the proposal is accepted as is, here are the blockers to solve.

### Issues (blockers)

- [ ] Create an `SCMaaS` interface and implement each supported providers. (github, gitlab, bitbucket server...)
- [ ] `coderd` and `coder agent` must be able to parse a devcontainer spec. (reuse `envbuilder` code ?)
- [ ] Add field `kind` in `coder_parameter`. See [example][secret-implem]
Values of `kind` are `git_branch`, `git_repository`, `devcontainer_config`, maybe `scm_provider` and `scm_project`
- [ ] Implement backend code of each value (using the interface/implems created before)
- [ ] Update Frontend - Proper validation + completion must be implemented.

### Devcontainer Issues (partially sorted by criticality)

- [ ] feat - [Devcontainer secrets support][secret-implem]
- [ ] feat - [Devcontainer Prebuilds](https://github.com/coder/coder/issues/5325#issuecomment-2227517151)
- [x] docs - How to make [dind feature](https://github.com/devcontainers/features/tree/main/src/docker-in-docker) works in any VM.
- [ ] IDEs Integration
- [ ] VSCode & Coder extension
- [ ] feat - dev feedback loop, detect changes, popup for rebuild container (+ usual buttons rebuild, full rebuild, view creation logs...)
- [ ] feat - Sync forwarded/shared ports between `vscode` <--> `web ui`
- [ ] feat - `vscode web` share ports trough url. (current workaround -> use `web ui` )
- [ ] Jetbrain IDEs & Coder extension
- [ ] feat - dev feedback loop (not sure how, check if GH have done it + then ask jetbrain if somehow their remote plugin could help )
- [ ] feat - [Make intelliJ aware it's running in devcontainer ](https://youtrack.jetbrains.com/issue/IJPL-180043/Devcontainer-Applying-json-settings-customization-in-a-CDE.)(for applying settings `customization.jetbrain.*` )
- [ ] feat - Reflect port configuration [from devcontainer](https://github.com/microsoft/vscode/blob/main/.devcontainer/devcontainer.json#L39) in coder UI.
- [ ] feat - fix chicken egg problem `devcontainer-cli` <-> `coder agent`, ( [see solution](https://github.com/coder/coder/issues/13358#issuecomment-2224080836))
Maybe a new cmd to makes it easier :
`CODER_TOKEN=... coder exec -- echo 'Agent not started yet but this appears in the UI.'`
- [ ] #15128


[devcontainer-cli]:https://github.com/devcontainers/cli
[secret-implem]:https://github.com/coder/coder/issues/7087#issuecomment-2417456949