Frogbot v2.24.1 changes behaviour for repositories with multiple Python packages
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 372
- Forks
- 107
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 5
Description
Describe the bug
Hi, after the release of v2.24.1, we're seeing a breaking change in behaviour when a single repository contains multiple nested packages.
The repositories affected have one top-level package, and multiple nested packages. Something like this:
repository/
├── requirements.txt
├── setup.py
├── submodule1
│ ├── requirements.txt
│ └── setup.py
└── submodule2
├── requirements.txt
└── setup.py
Current behavior
Previously Frogbot seems to only find and scan the top-level project (logs cut down for brevity):
Run jfrog/frogbot@v2
with:
version: latest
...
Frogbot
/home/runner/_work/_tool/frogbot/[RELEASE]/x64/frogbot scan-repository
14:59:26 [Info] Frogbot version: 2.24.0
14:59:26 [Info] Running Frogbot "scan-repository" command
14:59:31 [Info] Performing scans on 1 targets:
[
{
"target": "/tmp/jfrog.cli.temp.-1737471567-3618821560",
"technology": "pip"
}
]
But with 2.24.1, Frogbot now discovers the subprojects automatically and tries to scan them:
Run jfrog/frogbot@v2
with:
version: latest
...
::group::Frogbot
Frogbot
##[debug]Obtaining an access token through OpenID Connect...
##[debug]Fetching JSON web token
##[debug]ID token url is ...
::add-mask::***
##[debug]Exchanging GitHub JSON web token with a JFrog access token...
::add-mask::***
##[debug]Downloading Frogbot from https://releases.jfrog.io/artifactory/frogbot/v2/[RELEASE]/frogbot-linux-amd64/frogbot
##[debug]Downloading https://releases.jfrog.io/artifactory/frogbot/v2/[RELEASE]/frogbot-linux-amd64/frogbot
##[debug]Destination /home/runner/_work/_temp/218448a2-ac41-4100-a458-b1882a52f75f
##[debug]download complete
##[debug]Caching tool frogbot [RELEASE] x64
##[debug]source file: /home/runner/_work/_temp/218448a2-ac41-4100-a458-b1882a52f75f
##[debug]destination /home/runner/_work/_tool/frogbot/[RELEASE]/x64
##[debug]destination file /home/runner/_work/_tool/frogbot/[RELEASE]/x64/frogbot
##[debug]finished caching tool
/home/runner/_work/_tool/frogbot/[RELEASE]/x64/frogbot scan-pull-request
15:53:02 [Info] Frogbot version: 2.24.1
15:53:02 [Info] Running Frogbot "scan-pull-request" command
15:53:03 [Info] Scanning Pull Request #5899 (from source branch: <.../...> to target branch: <.../master>)
15:53:03 [Info] -----------------------------------------------------------
15:53:03 [Info] ... repository downloaded successfully. Starting with repository extraction...
15:53:06 [Info] Extracted repository successfully
15:53:06 [Info] Scanning source branch...
15:53:06 [Info] Performing scans on 3 targets:
[
{
"target": "/tmp/jfrog.cli.temp.-1737474783-193021820/submodule1",
"technology": "pip"
},
{
"target": "/tmp/jfrog.cli.temp.-1737474783-193021820/submodule1",
"technology": "pip"
},
{
"target": "/tmp/jfrog.cli.temp.-1737474783-193021820",
"technology": "pip"
},
]
This is a breaking change, because Frogbot's behaviour has changed without a major version increase. In our case, the scanner failed to build the submodules because they require more than the standard pip build to build successfully.
We get an error like this:
Error: 8 [Error] target '/tmp/jfrog.cli.temp.-1737474783-193021820/submodule1 [pip]' errors:
failed to build dependency tree: failed while building 'pip' dependency tree: "python -m pip install . -i ..." command failed: exit status 1 - Looking in indexes: ...
Followed by the usual pip build failure output.
Workaround: pin to 2.24.0 with:
- uses: jfrog/frogbot@v2.24.0
with:
version: "2.24.0"
Reproduction steps
Create a nested Python project repository as described above. Run Frogbot v2.24.1, then v2.24.0 on the same repository. Notice that the former discovers the subpackages, while the latter only discovers the top-level package.
Expected behavior
v2.24.1 should not include this breaking change in behaviour compared to v2.24.0.
This new behaviour should be put behind a config option until v3.
JFrog Frogbot version
2.24.1
Package manager info
pip 24.0, with setup.py
Git provider
GitHub
JFrog Frogbot configuration yaml file
jfrog.yaml
name: JFrog scan
on:
push:
branches:
- master
pull_request_target:
types: [opened, synchronize]
branches:
- master
schedule:
- cron: "0 0 * * 1-5"
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
scan:
runs-on: <omitted>
timeout-minutes: 30
permissions:
id-token: write
pull-requests: write
security-events: write
contents: read
steps:
- uses: actions/checkout@v4
- name: Check for Python files
id: python
run: |
if [ -f ".github/jfrog/.python-version" ]; then
echo "Found .github/jfrog/.python-version"
python_version=$(cat .github/jfrog/.python-version)
elif find . -name "*.py" 2>/dev/null | grep -q .; then
echo "Python files found"
sub_requirements=$(find . -path "**/requirements.txt" -not -path "./requirements.txt" | head -n 1)
if [ -f "pyproject.toml" ]; then
echo "Found pyproject.toml"
python_version=$(grep -E '^python' pyproject.toml || echo "No version found in pyproject.toml")
elif [ -f "requirements.txt" ]; then
echo "Found requirements.txt"
python_version=$(grep -oP 'This file is autogenerated by pip-compile with [Pp]ython \K[0-9]+\.[0-9]+' requirements.txt || echo '')
if [ -z "$python_version" ]; then
python_version="3.11"
fi
elif [ -n "$sub_requirements" ]; then
echo "Found requirements.txt in subdirectory: $sub_requirements"
python_version=$(grep -oP 'This file is autogenerated by pip-compile with [Pp]ython \K[0-9]+\.[0-9]+' "$sub_requirements" || echo '')
if [ -z "$python_version" ]; then
python_version="3.11"
fi
elif [ -f ".python-version" ]; then
python_version=$(cat .python-version)
else
python_version="Unknown"
fi
echo "Python version: $python_version"
else
python_version="Unknown"
echo "No Python files found"
fi
echo "python_version=$python_version" >> $GITHUB_OUTPUT
- name: Check for Node.js files
id: node
run: |
package_json_path=$(find . -name "package.json" | head -n 1)
sub_json_path=$(find . -name "package.json" -not -path "./package.json" | head -n 1)
if [ -n "$package_json_path" ]; then
echo "Node.js project found at $package_json_path"
node_found="true"
elif [ -n "$sub_json_path" ]; then
echo "Node.js project found at $sub_json_path"
node_found="true"
else
node_found="false"
echo "No Node.js files found"
fi
echo "node_found=$node_found" >> $GITHUB_OUTPUT
- uses: actions/setup-python@v5
if: ${{ steps.python.outputs.python_version != 'Unknown' }}
with:
python-version: ${{ steps.python.outputs.python_version }}
- uses: actions/setup-node@v4
if: ${{ steps.node.outputs.node_found == 'true' }}
with:
node-version: 22
- name: Install Yarn
if: ${{ steps.node.outputs.node_found == 'true' }}
run: |
npm install --global yarn
- name: Set JFrog base branch if master
if: github.event_name == 'schedule' || github.event_name == 'push'
run: echo "JF_GIT_BASE_BRANCH=master" >> $GITHUB_ENV
- name: Installation
if: ${{ hashFiles('.github/jfrog/pre_install.sh') != '' }}
run: |
./.github/jfrog/pre_install.sh
- name: Read Frogbot path exclusions
if: ${{ hashFiles('.github/jfrog/path_exclusions.txt') != '' }}
run: |
EXCLUSIONS="*git*;*node_modules*;*target*;*venv*;*test*;$(cat .github/jfrog/path_exclusions.txt)"
echo "JF_PATH_EXCLUSIONS=$EXCLUSIONS" >> $GITHUB_ENV
- uses: jfrog/frogbot@v2
with:
oidc-provider-name: github
oidc-audience: jfrog-github
env:
JF_DEPS_REPO: pypi-virtual
JF_GIT_TOKEN: ${{ secrets.GITHUB_TOKEN }}
JF_MIN_SEVERITY: Critical
JF_URL: <omitted>
# The Frogbot success comment is just a big, froggy banner advertising this tool--which we already pay for.
- name: Delete Frogbot success comment
if: github.event_name == 'pull_request_target'
uses: actions/github-script@v7
with:
script: |
const prNumber = context.payload.pull_request.number;
const comments = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber
});
for (const comment of comments.data) {
if (comment.body.includes("Frogbot scanned this pull request and did not find any new security issues")) {
await github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: comment.id
});
console.log(`Deleted comment ID: ${comment.id} (${context.repo.owner}/${context.repo.repo})`);
}
}
Operating system type and version
Ubuntu 22.04.5 LTS
JFrog Xray version
?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The affected entry points are scan-repository and scan-pull-request; begin by reproducing the nested setup.py and requirements.txt layout with Frogbot 2.24.1 and 2.24.0. Compare target discovery and configuration behavior, then verify that the default matches 2.24.0 or that the new discovery is gated by a configuration option.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, go, python
- Domain
- ci-cd, security, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100