dbt-labs / dbt-labs/dbt

[Implementation] Install agent skills via `dbt deps` using a new `skills:` key in `dependencies.yml`

Open
#12,868 0 comments 0 reactions 0 assignees View on GitHub
area:engine engine:v1 engine:v2 type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

Same thing as https://github.com/dbt-labs/dbt-core/issues/14556, but for Core.

### Housekeeping

- [x] I am a maintainer of dbt-core

### Short description

The idea is covered in this Discussion on GitHub:
- https://github.com/dbt-labs/dbt-core/discussions/12521

Add a `skills:` top-level key to `dependencies.yml`. When a user runs `dbt deps`, dbt installs the declared skills into the project alongside packages.

### Simple example

`dependencies.yml`

```
skills:
- git: "https://github.com/dbt-labs/dbt-agent-skills.git"
revision: 0.9.2
```

After `dbt deps`:
```
.agents/
└── skills/
├── adding-dbt-unit-test/
│ └── SKILL.md
└── running-dbt-commands/
└── SKILL.md
```

### Acceptance criteria

### Options
- `git` - Clone a git repo at an optional revision (tag, branch, or SHA)
- `local` - Copy from a local filesystem path (absolute or relative to project)
- `package` - download from the dbt Package Hub at a given version

### Optional fields
- `subdirectory` - Install from a subdirectory (rather than the root)
- `skills` - List of skill names to install (like an allowlist, skips any skills that aren't explicitly listed)
- `path` - Where to install (default `.agents/skills`). Useful for agents that don't support the `.agents/skills` default or the user wants to install globally instead of project-based

### Install behavior
- Runs as part of `dbt deps` (no new subcommand or flag)
- If it contains a top-level `skills/` directory, install from there
- Otherwise, if the source itself contains a `SKILL.md` at its root, install it as a single skill
- Otherwise, install each subdirectory that contains a `SKILL.md` (skipping the rest)
- Pre-existing skills in the destination that came from other sources are preserved (not wiped)
- Skills declared in `packages.yml` raise a clear error (because only allowed within `dependencies.yml`)

### Breadth of examples

`dependencies.yml`

```yaml
skills:

# dbt Hub
- package: dbt-labs/dbt_utils
version: 9.9.9

# git URL
- git: "https://github.com/dbt-labs/dbt-agent-skills.git"

# Local path
- local: /Users/dbeatty/jaffle-minis/minis/a244e_bare_minimum/nunchuck-skills

# Private GitHub URL
# TODO - not implemented yet -- see note below regarding using private repos with SSH
- private: your-org/your-repo

# explicit installation location - relative to the dbt project directory
- git: "https://github.com/dbt-labs/dbt-agent-skills.git"
revision: 65d2e0b68e24b59e038e6deb14fa6624c63022fe
path: ".agentZ/skills" # relative to the dbt project directory

# explicit installation location - absolute path
- git: "https://github.com/dbt-labs/dbt-agent-skills.git"
revision: 65d2e0b68e24b59e038e6deb14fa6624c63022fe
path: "~/.agents/skills" # absolute path

# multiple installation locations
- git: "https://github.com/dbt-labs/dbt-agent-skills.git"
path:
- ".agents/skills"
- ".codex/skills"

# `revision` for git tag, branch name, or SHA (40-character hash)
- git: "https://github.com/dbt-labs/dbt-agent-skills.git"
revision: 0.1.0 # git tag
# revision: dbeatty/feature-branch # branch name
# revision: 65d2e0b68e24b59e038e6deb14fa6624c63022fe # SHA (40-character hash)

# `subdirectory` within the source directory
- local: /Users/dbeatty/jaffle-minis/minis/a244e_bare_minimum
subdirectory: "nunchuck-skills" # name of subdirectory containing one or skills

# skills to install (YAML list syntax 1)
- git: "https://github.com/dbt-labs/dbt-agent-skills.git"
skills:
- building-dbt-semantic-layer

# skills to install (YAML list syntax 2)
- git: "https://github.com/dbt-labs/dbt-agent-skills.git"
skills: ["troubleshooting-dbt-job-errors"]

# Case 1: source is a Skill directory
- local: /Users/dbeatty/jaffle-minis/minis/a244e_bare_minimum/skills/dinomight

# Case 2: source contains one or more Skill directories
- local: /Users/dbeatty/jaffle-minis/minis/a244e_bare_minimum/skills/

# Case 3: source contains a `skills/` directory (that contains Skill subdirectories)
- local: /Users/dbeatty/jaffle-minis/minis/a244e_bare_minimum/

# Further testing - with `subdirectory`

# Case 1 - with `subdirectory`
- local: /Users/dbeatty/jaffle-minis/minis/a244e_bare_minimum/skills
subdirectory: dinomight

# Case 2 - with `subdirectory`
- local: /Users/dbeatty/jaffle-minis/minis/a244e_bare_minimum/
subdirectory: skills/

# Case 3 - with `subdirectory`
- local: /Users/dbeatty/jaffle-minis/minis/
subdirectory: a244e_bare_minimum
```

### Suggested Tests

- `dbt deps` with only a `skills:` entry in `dependencies.yml` (no `packages:` specificed) still installs the skills
- `dbt deps` installs skills from each source type: `git`, `local`, `package`
- Default of `.agents/skills` is used when `path` is not specified
- `path` as a string installs only to that one location
- `path` as a list installs to each location listed
- `subdirectory` override is utilized and takes precedence over the root
- `skills: ["a", "b"]` installs only these skills and skips all others
- A pre-existing skill folder in `.agents/skills/` is preserved after running `dbt deps`
- A source whose root is itself a skill (`SKILL.md` at root) installs as a single named folder
- Folders that do not contain a `SKILL.md` are ignored (not installed)
- A bad `package:` name raises a clear "not found in the package index" error (not a connection traceback)
- Defining `skills:` inside `packages.yml` raises a `DbtProjectError`

### Impact to Other Teams

- Docs: needs documentation for `skills:` in `dependencies.yml` -- existing docs [here](https://docs.getdbt.com/docs/mesh/govern/project-dependencies).
- Package Hub: no changes required since a skill package is just a repo with a `skills/` directory or a root-level `SKILL.md`. But for a package to be on the Hub, it _does_ need a `dbt_project.yml` at minimum.

### Will backports be required?

Nope.

### Context

[This discussion](https://github.com/dbt-labs/dbt-core/discussions/12521) included several pieces we chose to leave out:
- `agents:` -- the feedback in the discussion was that adding per-agent config creates complexity that we can/should avoid. Instead, we'll install to `.agents/skills` by default and offer `path` as the escape hatch.
- `scope: global`-- same thing. Skills are project-scoped by default. Users who want a skill available in every project can use the `path` option or install globally on their own.
- `dbt_project.yml` configuration for skills (`enabled`, `meta`, `tags`, `skills-paths`) -- not needed for the initial implementation. If we discover use cases later, we can expand at that point.
- `private:` GitHub URLs via HTTPS or SSH -- this will surely be crucial for many users, so it is definitely worth a follow-up in a separate issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.