Devcontainer Integration (coderd, agent, ui and IDEs)

Coder was the first alternative to `codespaces`.
But as of Oct. 2024 it lack behind `gitpod` and `daytona` in term of `devcontainer` support.
IMHO this is a critical issue that should be fixed.

### Context

Since the first release of `envbuilder` in sept 2023 we got a bunch of really nice releases notes. 🚀
But none included a mention of some form of devcontainer integration with `coder`. 💔

Coder's concurrents don't care about re-implementing the whole `devcontainer spec`.
Instead they just use the [devcontainer cli][devcontainer-cli] and focus their time on the `UX` and `integration`.

The only advantage of `envbuilder` I see is for running workspaces as pod, that's it.
For workspaces as VM `envbuilder` currently does not make any sense to me.
The [official implem][devcontainer-cli] is currently much better because it implements the whole spec.
(prebuilds, secrets, merge configs (features, extensions, etc..), output effective conf, lock file, dotfiles, compose...)

As a user I really like coder's flexibility, but the devcontainer DevX is just bad.
It's makes me reconsider less flexible alternatives like `daytona`.

I hope `coder` will focus now on filling the feature parity gap with `daytona`, `gitpod` and hopefully `codespaces`.

---

### Implementation Proposal

In a similar way github do, `coder` should query the SCMaaS provider to know which branches exists and fetch the current configuration.


In short `coderd` and the `agent` must be able to parse a devcontainer config.
Respectively before `provisioning time` and during `runtime`.

Before the workspace creation form is submitted `coderd` must fetch and parse the devcontainer config. (for features like [secrets][secret-implem], prebuilds, hardware requirements... )

#### How

1. Add an `SCM provider` interface + implement each supported providers ( github, gitlab, bitbucket server...)
2. Add a new field `kind` in `coder_parameter`. (when `kind` is set `validate` field become useless)
Each values bind to it's own method in `coderd`.
Theses methods will autocomplete / validate the `coder_parameter` through websockets. (they'll use the scm interface for fetching data)
Config example :
```yaml
data "coder_parameter" "repository_url" {
name = "repository_url"
display_name = "Repository URL"
default = "https://hostname/.../repo.git"
mutable = true
# The server would use this field to know it's the repository url.
kind = "git_repository"
# It could also guess the coder_external_auth with the hostname in the url.
# Not a great UX though, no drop down list of projects and repositories...
# Could use three separate fields instead, ex: scm_provider (gitlab, bitbucket...), scm_project and scm_repository
}
data "coder_parameter" "target_branch" {
name = "target_branch"
display_name = "Branch"
mutable = true
kind = "git_branch" # point to some server side code that will query the git provider
}
data "coder_parameter" "devcontainer_config" {
name = "devcontainer_config"
display_name = "Devcontainer configuration"
default = ".devcontainer/devcontainer.json" # ignored anyway, file may not exists
mutable = true
kind = "devcontainer_config" # coderd will complete / validate and fetch/parse the file right away for secrets and other features)
}
# we could add other values for validating `hostRequirements`... ram, cpu, storage, GPUs...
# eg: devcontainer_ram, devcontainer_cpu, devcontainer_storage...
```

First it will improve the DevX with completion / validation on fields like `branch` , `devcontainer config`.
And then enable implementing important features such as [devcontainer secrets][secret-implem] and `prebuilds`...

#### Going further

The proposal above is quite flexible and could be reused outside of `devcontainers`.

The SCM provider could be extended for other use-cases.
As example it could be reused for features related to gitops and the new [coderd tf provider](https://github.com/coder/terraform-provider-coderd).
It could show infos of the gitops repo(s) right in the UI or CLI. (taking inspiration from `argocd` )
It also enable fetching a config file like `coder.yaml` from a repository to configure features like prebuilds.

The `kind` field can also be used for other purposes.
We could think of a way to let template admins define their own custom `values` and `methods` to validate `coder_parameters`.
Some examples would be validate / autocomplete :
- An artifact path from `artifactory` or equivalent.
- Selecting data from k8s or any cloud provider api
- An instance_type or regions
- Selecting a deployed resource name.
- Validating that a resource name is available or not.
- ?

Any feedback / enhancements welcomed, feel free to reach out to me on slack/discord
Happy to help

---

If the proposal is accepted as is, here are the blockers to solve.

### Issues (blockers)

- [ ] Create an `SCMaaS` interface and implement each supported providers. (github, gitlab, bitbucket server...)
- [ ] `coderd` and `coder agent` must be able to parse a devcontainer spec. (reuse `envbuilder` code ?)
- [ ] Add field `kind` in `coder_parameter`. See [example][secret-implem]
Values of `kind` are `git_branch`, `git_repository`, `devcontainer_config`, maybe `scm_provider` and `scm_project`
- [ ] Implement backend code of each value (using the interface/implems created before)
- [ ] Update Frontend - Proper validation + completion must be implemented.

### Devcontainer Issues (partially sorted by criticality)

- [ ] feat - [Devcontainer secrets support][secret-implem]
- [ ] feat - [Devcontainer Prebuilds](https://github.com/coder/coder/issues/5325#issuecomment-2227517151)
- [x] docs - How to make [dind feature](https://github.com/devcontainers/features/tree/main/src/docker-in-docker) works in any VM.
- [ ] IDEs Integration
- [ ] VSCode & Coder extension
- [ ] feat - dev feedback loop, detect changes, popup for rebuild container (+ usual buttons rebuild, full rebuild, view creation logs...)
- [ ] feat - Sync forwarded/shared ports between `vscode` <--> `web ui`
- [ ] feat - `vscode web` share ports trough url. (current workaround -> use `web ui` )
- [ ] Jetbrain IDEs & Coder extension
- [ ] feat - dev feedback loop (not sure how, check if GH have done it + then ask jetbrain if somehow their remote plugin could help )
- [ ] feat - [Make intelliJ aware it's running in devcontainer ](https://youtrack.jetbrains.com/issue/IJPL-180043/Devcontainer-Applying-json-settings-customization-in-a-CDE.)(for applying settings `customization.jetbrain.*` )
- [ ] feat - Reflect port configuration [from devcontainer](https://github.com/microsoft/vscode/blob/main/.devcontainer/devcontainer.json#L39) in coder UI.
- [ ] feat - fix chicken egg problem `devcontainer-cli` <-> `coder agent`, ( [see solution](https://github.com/coder/coder/issues/13358#issuecomment-2224080836))
Maybe a new cmd to makes it easier :
`CODER_TOKEN=... coder exec -- echo 'Agent not started yet but this appears in the UI.'`
- [ ] #15128


[devcontainer-cli]:https://github.com/devcontainers/cli
[secret-implem]:https://github.com/coder/coder/issues/7087#issuecomment-2417456949

No files or tests are named. Start by mapping the coderd, agent, frontend, coder_parameter, envbuilder, and SCM provider entry points, then compare the proposed workflow with the devcontainer CLI. Done would require an agreed scope and integrated SCM, devcontainer, parameter validation, and IDE behavior rather than one isolated change.

Written by the indexing model from the issue text.