[CT-3507] [Bug] Installing packages to cache location fails with "Cannot call rmtree on a symbolic link"
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Is this a new bug in dbt-core?
- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug
### Current Behavior
Running `dbt deps` in AWS CodeBuild results in this error:
> Cannot call rmtree on a symbolic link
On this line: https://github.com/dbt-labs/dbt-core/blame/2401600e57048dd56818f7293abed96ffd510ac9/core/dbt/task/deps.py#L231
Notable observation is that we use CodeBuild caching for dbt packages.
### Expected Behavior
Not fail.
### Steps To Reproduce
1. Run dbt 1.7.0 in CodeBuild
2. Store packages in cache (`/root/.cache/dbt_packages`)
3. Run `dbt deps`
### Relevant log output
```shell
[
{
"data": {
"log_version": 3,
"version": "=1.7.3"
},
"info": {
"category": "",
"code": "A001",
"extra": {},
"invocation_id": "454cd6b0-139a-47fc-8d90-52dd3e52bf5a",
"level": "info",
"msg": "Running with dbt=1.7.3",
"name": "MainReportVersion",
"pid": 231,
"thread": "MainThread",
"ts": "2023-12-19T11:02:05.648499Z"
}
},
{
"data": {
"exc": "Cannot call rmtree on a symbolic link"
},
"info": {
"category": "",
"code": "Z002",
"extra": {},
"invocation_id": "454cd6b0-139a-47fc-8d90-52dd3e52bf5a",
"level": "error",
"msg": "Encountered an error:\nCannot call rmtree on a symbolic link",
"name": "MainEncounteredError",
"pid": 231,
"thread": "MainThread",
"ts": "2023-12-19T11:02:06.100657Z"
}
},
{
"data": {
"stack_trace": "Traceback (most recent call last):\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/cli/requires.py\", line 90, in wrapper\n result, success = func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/cli/requires.py\", line 75, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/cli/requires.py\", line 151, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/cli/requires.py\", line 197, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/cli/main.py\", line 492, in deps\n results = task.run()\n ^^^^^^^^^^\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/task/deps.py\", line 228, in run\n system.rmtree(self.project.packages_install_path)\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/clients/system.py\", line 570, in rmtree\n return shutil.rmtree(path, onerror=chmod_and_retry)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/root/.pyenv/versions/3.11.6/lib/python3.11/shutil.py\", line 744, in rmtree\n onerror(os.path.islink, path, sys.exc_info())\n File \"/root/.pyenv/versions/3.11.6/lib/python3.11/shutil.py\", line 742, in rmtree\n raise OSError(\"Cannot call rmtree on a symbolic link\")\nOSError: Cannot call rmtree on a symbolic link\n"
},
"info": {
"category": "",
"code": "Z003",
"extra": {},
"invocation_id": "454cd6b0-139a-47fc-8d90-52dd3e52bf5a",
"level": "error",
"msg": "Traceback (most recent call last):\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/cli/requires.py\", line 90, in wrapper\n result, success = func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/cli/requires.py\", line 75, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/cli/requires.py\", line 151, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/cli/requires.py\", line 197, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/cli/main.py\", line 492, in deps\n results = task.run()\n ^^^^^^^^^^\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/task/deps.py\", line 228, in run\n system.rmtree(self.project.packages_install_path)\n File \"/root/.cache/venv/lib/python3.11/site-packages/dbt/clients/system.py\", line 570, in rmtree\n return shutil.rmtree(path, onerror=chmod_and_retry)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/root/.pyenv/versions/3.11.6/lib/python3.11/shutil.py\", line 744, in rmtree\n onerror(os.path.islink, path, sys.exc_info())\n File \"/root/.pyenv/versions/3.11.6/lib/python3.11/shutil.py\", line 742, in rmtree\n raise OSError(\"Cannot call rmtree on a symbolic link\")\nOSError: Cannot call rmtree on a symbolic link\n",
"name": "MainStackTrace",
"pid": 231,
"thread": "MainThread",
"ts": "2023-12-19T11:02:06.178555Z"
}
}
]
```
### Environment
```markdown
- OS: Ubuntu 22.04 (`aws/codebuild/standard:7.0`)
- Python: 3.11
- dbt: 1.7.3
```
### Which database adapter are you using with dbt?
_No response_
### Additional Context
Using cached location for dbt packages.
I've tried narrowing it down, and the issue starts happening on `1.7.0` release.
Contributor guide
Assessment
This issue has not been assessed yet.