dotnet / dotnet/arcade

Job template uploads logs multiple times.

Open
#15,999 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
729
Forks
397
Avg merge
3d 15m
Merged PRs (30d)
149

Description

When using the jobs template, the logs are being uploaded/stored multiple times.

``` yml
- template: ${{ parameters.engCommonTemplatesDir }}/jobs/jobs.yml
parameters:
artifacts:
publish:
artifacts: true
logs: true
```

Illustration of a build with the above yml. The two folders within the red boxes have the same contents.

Image

This appears to be caused by the gather logic putting everything in subfolder of an artifacts directory as shown below.

https://github.com/dotnet/arcade/blob/0c4027923c24fb5cd00ac8ccbb2e42b0f3fcff40/eng/common/core-templates/job/job.yml#L183C1-L206C28
``` csharp
# gather artifacts
- ${{ if ne(parameters.artifacts.publish, '') }}:
- ${{ if and(ne(parameters.artifacts.publish.artifacts, 'false'), ne(parameters.artifacts.publish.artifacts, '')) }}:
- task: CopyFiles@2
displayName: Gather binaries for publish to artifacts
inputs:
SourceFolder: 'artifacts/bin'
Contents: '**'
TargetFolder: '$(Build.ArtifactStagingDirectory)/artifacts/bin'
- task: CopyFiles@2
displayName: Gather packages for publish to artifacts
inputs:
SourceFolder: 'artifacts/packages'
Contents: '**'
TargetFolder: '$(Build.ArtifactStagingDirectory)/artifacts/packages'
- ${{ if and(ne(parameters.artifacts.publish.logs, 'false'), ne(parameters.artifacts.publish.logs, '')) }}:
- task: CopyFiles@2
displayName: Gather logs for publish to artifacts
inputs:
SourceFolder: 'artifacts/log'
Contents: '**'
TargetFolder: '$(Build.ArtifactStagingDirectory)/artifacts/log'
continueOnError: true
condition: always()
```

...and the publish logic for the artifacts publishes the entire artifacts directory.

https://github.com/dotnet/arcade/blob/0c4027923c24fb5cd00ac8ccbb2e42b0f3fcff40/eng/common/templates/job/job.yml#L37C1-L61C64
```
- ${{ if ne(parameters.artifacts.publish, '') }}:
- ${{ if and(ne(parameters.artifacts.publish.artifacts, 'false'), ne(parameters.artifacts.publish.artifacts, '')) }}:
- template: /eng/common/core-templates/steps/publish-build-artifacts.yml
parameters:
is1ESPipeline: false
args:
displayName: Publish pipeline artifacts
pathToPublish: '$(Build.ArtifactStagingDirectory)/artifacts'
publishLocation: Container
artifactName: ${{ coalesce(parameters.artifacts.publish.artifacts.name , 'Artifacts_$(Agent.Os)_$(_BuildConfig)') }}
continueOnError: true
condition: always()
retryCountOnTaskFailure: 10 # for any logs being locked
- ${{ if and(ne(parameters.artifacts.publish.logs, 'false'), ne(parameters.artifacts.publish.logs, '')) }}:
- template: /eng/common/core-templates/steps/publish-pipeline-artifacts.yml
parameters:
is1ESPipeline: false
args:
targetPath: '$(Build.ArtifactStagingDirectory)/artifacts/log'
artifactName: ${{ coalesce(parameters.artifacts.publish.logs.name, 'Logs_Build_$(Agent.Os)_$(_BuildConfig)') }}
displayName: 'Publish logs'
continueOnError: true
condition: always()
retryCountOnTaskFailure: 10 # for any logs being locked
sbomEnabled: false # we don't need SBOM for logs
```

Contributor guide

Open the contributing guide

Research direction

Start with eng/common/core-templates/job/job.yml around the gather-artifacts steps, then compare it with eng/common/templates/job/job.yml and the referenced publish templates. Trace the artifacts and logs paths in the jobs template configuration shown in the issue. Done means a build using both publish.artifacts and publish.logs stores each log only once.

Written by the indexing model from the issue text.

Assessment

Tech stack
yaml
Domain
build-system, ci-cd
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.