dbt-labs / dbt-labs/dbt

[Bug] uninstalled package check compares counts instead of the package names it works out

Open
#16,179 0 comments 0 reactions 0 assignees View on GitHub
triage
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

_edited twice: my first version described the problem wrongly, and the second was hard to read. this version is the same report in plainer words._

### is this a new bug in dbt-core?
- [x] i believe this is a new bug in dbt-core
- [x] i have searched the existing issues, and i could not find an existing issue for this bug

### current behavior

#12508 says it changed this check to compare package names:

> Instead of comparing package counts from packages.yml we will compare the actual package names from `package-lock.yml` with the installed package's directory names and check if there are any locked packages that are not installed.

the code that shipped works out that list of names, but never checks it. it compares counts instead (`core/dbt/config/runtime.py`, v1.12.3):

```python
count_packages_specified = len(specified_packages)
count_packages_installed = len(installed_package_names)

uninstalled_packages = specified_package_names - installed_package_names

# we expect same number of packages specified and installed
if count_packages_specified != count_packages_installed:
raise UninstalledPackagesFoundError(...)
```

the test is `count_packages_specified != count_packages_installed`. `uninstalled_packages` is only used to write the error message.

for reference, 1.11.7 tested the same two counts with `>`:

```python
if count_packages_specified > count_packages_installed:
```

so a project with more packages on disk than it asks for passed in 1.11 and fails in 1.12.

this causes two problems.

**1. the check fails when nothing is missing.** if `dbt_packages` holds more packages than the project needs, the two counts differ and dbt stops. every package the project asked for is present. the error says packages are not installed when they are.

**2. the message often cannot name anything.** if `package-lock.yml` does not load, `specified_packages` falls back to `self.packages.packages`. but `specified_package_names` is still read from `locked_packages.packages`, which is empty on that path. so the set of missing names is empty too, the "Following packages were not found:" line is skipped, and you get two numbers and nothing else.

we hit both at once:

```
Compilation Error
dbt expects 6 package(s) based on packages specified in packages.yml, but found only 10 package(s) installed in dbt_packages. Run "dbt deps" to install package dependencies.
```

6 is the number of packages in our `packages.yml`. our lock file lists 8, so this is the fallback path. 10 is the number of folders in `dbt_packages`. no package is named, so we cannot tell from this message whether a package was really missing. both 6 and 8 differ from 10, so the check fails either way. "found only 10" also reads oddly when 10 is more than 6.

### expected behavior

fail on what the pr describes: a locked package that is not installed. do not fail when there are extra packages on disk. drop the counts from the message and say which packages are missing:

```
Compilation Error
these packages are in package-lock.yml but are not installed in dbt_packages:
- dbt_date
Run "dbt deps" to install package dependencies.
```

and read the list of names from the same source the counts came from, so the fallback path can name them too.

### steps to reproduce

1. take any project with a `packages.yml`
2. run `dbt deps`
3. add one extra package folder to `dbt_packages`, and remove nothing
4. run `dbt parse`

it fails, even though no locked package is missing.

### relevant log output

```shell
08:16:09 [ERROR]: Encountered an error:
Compilation Error
dbt expects 6 package(s) based on packages specified in packages.yml, but found only 10 package(s) installed in dbt_packages. Run "dbt deps" to install package dependencies.
```

### environment

```markdown
- OS: ubuntu, github actions runner
- Python: 3.12
- dbt: dbt-core 1.12, dbt-bigquery 1.12
```

### which database adapter are you using with dbt?
bigquery

### additional context

we found this when our ci moved from 1.11 to 1.12. our runner image builds one `dbt_packages` and shares it across several dbt projects, so `dbt deps` does not run for each project. the image installs more packages than any single project asks for, so the counts never match. this was fine in 1.11. in 1.12 every build fails.

we think #12508 is a good change and are not asking for it to be undone. the code just does not do what the pr says it does.

related: #12509, #12508, #10760, #16180

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.