jfrog / jfrog/jfrog-cli-security

Fail to upload Python dependencies for Xray scan in pipBuild

Open
#441 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
13
Forks
50
Avg merge
3d 48m
Merged PRs (30d)
26

Description

Describe the bug
When we use Artifactory plugin to build python script and upload the package to Artifactory. Zero dependencies were uploaded

To Reproduce

  1. Create Python type local repo on artifactory server: Python-Test

  2. Create Jenkins job to install and upload python packages as below

     stage('setup') { 
         steps {
            	script {
     			echo "artifactory Server name = ${artifactoryServer}"
            		rtServer = Artifactory.server "${artifactoryServer}"
                 
            		rtPip = Artifactory.newPipBuild()
            		rtPip.resolver repo: appJson.virtualName, server: rtServer
            		
       			buildInfo = Artifactory.newBuildInfo()
     	        buildInfo.env.capture = true
     	        scanConfig = [
     	            'buildName'      : buildInfo.name,
     	            'buildNumber'    : buildInfo.number,
     	            'failBuild'      : false
     	        ]
    			}
     	}
     }
     
     stage('VirtualEnv') {
         steps {
             dir("${deployLocation}app/${repo}") {
                 script {
     				virtual_env_activation = 'source activate "${repo}"'
     				sh'''                        
     				pwd
                     find . -iname "requirements.txt" | more
     				'''
     				requirements = readFile "./requirements.txt"
     				echo "requirements.txt = ${requirements}"
     				rtPip.install buildInfo: buildInfo, args: '-r "./requirements.txt"', envActivation: virtual_env_activation
             	}
             }
         }	
     }
    

    stage ('Package and create distribution archives') {
    steps {
    dir("${deployLocation}app/${repo}") {
    script {
    sh '''
    source activate "${repo}"
    python setup.py sdist bdist_wheel
    '''
    }
    }
    }
    }

    stage ('Upload packages') {
    steps {
    dir("${deployLocation}app/${repo}") {
    script {
    def uploadSpec = """{
    "files": [
    {
    "pattern": "./dist/",
    "target": "Python-Test/package/"
    }
    ]
    }"""
    rtServer.upload buildInfo: buildInfo, spec: uploadSpec
    }
    }
    }
    }

         stage('PublishBuildInfo') {
       	steps {
         	script {
     	    	rtServer.publishBuildInfo buildInfo
    			}
     	}
     }
    

After the package uploaded, zero dependencies were uploaded.

Below is the python requirements.txt file:
PyYAML==5.4.1
pytest==6.1.2
pytest-cov==2.11.1
requests==2.18.4
dse-tools==1.1.0
pandas==0.20.3
psutil==5.7.3
xlrd==1.2.0
openpyxl==3.0.5
pylint==2.6.0
jsonschema==3.2.0
feather-format==0.4.1
pyarrow==0.17.1

Expected behavior
We are building Python script on jenkins slave agent server. We have already installed and setup the python environment. All of the dependencies have already been downloaded into jenkins slave server. This is not a best practice to download dependency from artifactory every time when we build python script. We hope the plugin could upload the reqirements.txt to artifactory server to allow XRay to scan the dependencies.

Screenshots
If applicable, add screenshots to help explain your problem.
image

Versions

  • Jenkins Artifactory plugin version: 3.10.6
  • Jenkins operating system: Red Hat Enterprise Linux Server release 7.8 (Maipo)
  • Artifactory Version: 6.16.0

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Jenkins pipeline with Artifactory plugin 3.10.6, focusing on rtPip.install, buildInfo, and the upload and publishBuildInfo stages. Verify whether locally installed requirements are recorded in build information for Xray scanning. Done means the listed dependencies are associated with the published build without requiring a fresh download.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, devops, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.