Job template uploads logs multiple times.
- Dominant language
- C#
- Stars
- 729
- Forks
- 397
- Avg merge
- 3d 15m
- Merged PRs (30d)
- 149
Description
When using the jobs template, the logs are being uploaded/stored multiple times.
``` yml
- template: ${{ parameters.engCommonTemplatesDir }}/jobs/jobs.yml
parameters:
artifacts:
publish:
artifacts: true
logs: true
```
Illustration of a build with the above yml. The two folders within the red boxes have the same contents.
This appears to be caused by the gather logic putting everything in subfolder of an artifacts directory as shown below.
https://github.com/dotnet/arcade/blob/0c4027923c24fb5cd00ac8ccbb2e42b0f3fcff40/eng/common/core-templates/job/job.yml#L183C1-L206C28
``` csharp
# gather artifacts
- ${{ if ne(parameters.artifacts.publish, '') }}:
- ${{ if and(ne(parameters.artifacts.publish.artifacts, 'false'), ne(parameters.artifacts.publish.artifacts, '')) }}:
- task: CopyFiles@2
displayName: Gather binaries for publish to artifacts
inputs:
SourceFolder: 'artifacts/bin'
Contents: '**'
TargetFolder: '$(Build.ArtifactStagingDirectory)/artifacts/bin'
- task: CopyFiles@2
displayName: Gather packages for publish to artifacts
inputs:
SourceFolder: 'artifacts/packages'
Contents: '**'
TargetFolder: '$(Build.ArtifactStagingDirectory)/artifacts/packages'
- ${{ if and(ne(parameters.artifacts.publish.logs, 'false'), ne(parameters.artifacts.publish.logs, '')) }}:
- task: CopyFiles@2
displayName: Gather logs for publish to artifacts
inputs:
SourceFolder: 'artifacts/log'
Contents: '**'
TargetFolder: '$(Build.ArtifactStagingDirectory)/artifacts/log'
continueOnError: true
condition: always()
```
...and the publish logic for the artifacts publishes the entire artifacts directory.
https://github.com/dotnet/arcade/blob/0c4027923c24fb5cd00ac8ccbb2e42b0f3fcff40/eng/common/templates/job/job.yml#L37C1-L61C64
```
- ${{ if ne(parameters.artifacts.publish, '') }}:
- ${{ if and(ne(parameters.artifacts.publish.artifacts, 'false'), ne(parameters.artifacts.publish.artifacts, '')) }}:
- template: /eng/common/core-templates/steps/publish-build-artifacts.yml
parameters:
is1ESPipeline: false
args:
displayName: Publish pipeline artifacts
pathToPublish: '$(Build.ArtifactStagingDirectory)/artifacts'
publishLocation: Container
artifactName: ${{ coalesce(parameters.artifacts.publish.artifacts.name , 'Artifacts_$(Agent.Os)_$(_BuildConfig)') }}
continueOnError: true
condition: always()
retryCountOnTaskFailure: 10 # for any logs being locked
- ${{ if and(ne(parameters.artifacts.publish.logs, 'false'), ne(parameters.artifacts.publish.logs, '')) }}:
- template: /eng/common/core-templates/steps/publish-pipeline-artifacts.yml
parameters:
is1ESPipeline: false
args:
targetPath: '$(Build.ArtifactStagingDirectory)/artifacts/log'
artifactName: ${{ coalesce(parameters.artifacts.publish.logs.name, 'Logs_Build_$(Agent.Os)_$(_BuildConfig)') }}
displayName: 'Publish logs'
continueOnError: true
condition: always()
retryCountOnTaskFailure: 10 # for any logs being locked
sbomEnabled: false # we don't need SBOM for logs
```
Contributor guide
Research direction
Start with eng/common/core-templates/job/job.yml around the gather-artifacts steps, then compare it with eng/common/templates/job/job.yml and the referenced publish templates. Trace the artifacts and logs paths in the jobs template configuration shown in the issue. Done means a build using both publish.artifacts and publish.logs stores each log only once.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- yaml
- Domain
- build-system, ci-cd
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100