cake-build / cake-build/cake

Cake Frosting - cancelling build within Azure

Open
#4,014 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
4.2k
Forks
778
Avg merge
1h 15m
Merged PRs (30d)
19

Description

### Prerequisites

- [X] I have written a descriptive issue title
- [X] I have searched [issues](https://github.com/cake-build/cake/issues) to ensure it has not already been reported

### Cake runner

Cake Frosting

### Cake version

2.0.0

### Operating system

Windows

### Operating system architecture

64-Bit

### CI Server

_No response_

### What are you seeing?

I have made by own custom launcher of devenv.com - to be able to build solution.

It works as expected - it launches Visual studio's devenv.com and compiles entire solution.

However - during compilation we have some amount of C++ projects, which are compiling as in parallel and eating a lot of CPU.

When C++ compilation is in progress - and you try to cancel build job from azure - cakebuilder is cancelled, but devenv.com remains running.

What I see from log files - cakebuilder receives cancel signal in form of Ctrl+C or Ctrl+Break (seeing from logs), but result is that cakebuild is cancelled / killed / terminated, but devenv.com remains running.

I think this has something to do with C++ parallel compilation, as 8 CPU's are working on compiling C++, and maybe because of that cakebuilder does not gets execution priority to actually cancel devenv.com. I have tested this locally and manually pressed Ctrl+C - everything works correctly. Maybe on Azure's there is faster timeout happening, and because of that azure decides to kill cakebuilder without terminating it's child compilation processes.

This could be also classified as bug of Azure itself, as from my perspective it should also kill child processes, not only main builder process.

For now I have coded somehow similar `killbuildtasks.ps1`:

```
param (
[string]$categories = "build,test",

[Alias('v')]
[switch]$verbose=$false
)

$procs2kill = New-Object System.Collections.Generic.List[System.Object]

foreach( $category in $categories.Split(",;".ToCharArray()) )
{
if($category -eq 'build')
{
$procs2kill.AddRange(
@(
"devenv.com",
# launched by devenv.com
"devenv.exe",
"msbuild.exe",
"cmake.exe",
"ninja.exe",
"cl.exe",
"link.exe",
"vcpkg.exe",
"mspdbsrv.exe"
)
)
}

if($category -eq 'test')
{
$procs2kill.AddRange(
@(
"vstest.console.exe",
"testhost.x86.exe",
"testhost.exe",
"testhost.net472.exe"
)
)
}
}

# $procs2kill.Add('notepad.exe')

if($verbose -eq $true)
{
"Searching following processes:"
$procs2kill
}

Get-Process | ForEach-Object {

if([string]::IsNullOrEmpty($_.Path))
{
# Ignore where we cannot get path
return
}

$file = [System.IO.Path]::GetFileName($_.Path)
if( $procs2kill.Contains($file) )
{
# Don't kill Visual studio instances, but if building solution
if( $file -eq 'devenv.exe')
{
$query = "processid = '" + $_.Id + "'"
$process = Get-CimInstance Win32_Process -Filter $query | Select-Object CommandLine
$cmdLine = $process.CommandLine
if(-not $cmdLine.Contains('/build') )
{
return
}
}

"Killing $file (" + $_.Id + ")"

# Does not always works
#Stop-Process -Id $_.Id -Force

$cmd = "cmd.exe"
$cmdArgs = @( '/c', 'wmic', 'process', 'where', ('processid="' + $_.Id + '"'), 'call', 'terminate', '>', 'nul')

#$cmdArgs
& $cmd $cmdArgs
}
}

```

and added into yaml something like this:
```
- script: |
@echo on
powershell -executionpolicy bypass -file killbuildtasks.ps1
displayName: Kill build tasks
condition: always()
```

But this feels like a crack solution.

Maybe someone can propose a better solution on this one ?

One approach could be:
- Put cake builder itself to higher priority over child processes, so it can process cancel signal faster, and would not end up into process termination.

### What is expected?

All child compilation tasks gets terminated before terminating main frosten cake builder task.

### Steps to Reproduce

Really difficult to reproduce - need to have heavy c++ project with cakebuilder adapted to it.
Need to run cake builder task in azure.
Perform cancel in Azure while compilation is up and running.

Cannot provide my own build environment due to privacy of the project.

### Output log

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with Cake Frosting's cancellation path for the Azure Ctrl+C or Ctrl+Break scenario and compare it with the supplied killbuildtasks.ps1 workaround and local behavior. The issue names no repository files or tests; done means reproducing cancellation during the heavy C++ build and verifying whether devenv.com and its child compilation processes terminate.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, powershell
Domain
build-system, ci-cd, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.