fsprojects / fsprojects/Paket

dotnet restore segfaults when running in Azure pipeline

Open
#3,628 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
F#
Stars
2.1k
Forks
528
Avg merge
1d 12m
Merged PRs (30d)
54

Description

Description

Paket version: 5.215.0

When using paket with .NET Core in an Azure pipeline (on Hosted Ubuntu 1604), dotnet restore fails in a number of different ways (including segfaults).

The Hosted Ubuntu 1604 agent provides the following environment:

+ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.2.105
 Commit:    7cecb35b92

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  16.04
 OS Platform: Linux
 RID:         ubuntu.16.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.2.105/

Host (useful for support):
  Version: 2.2.3
  Commit:  6b8ad509b6

.NET Core SDKs installed:
  1.0.4 [/usr/share/dotnet/sdk]
  1.1.4 [/usr/share/dotnet/sdk]
  1.1.5 [/usr/share/dotnet/sdk]
  1.1.7 [/usr/share/dotnet/sdk]
  1.1.8 [/usr/share/dotnet/sdk]
  1.1.9 [/usr/share/dotnet/sdk]
  1.1.10 [/usr/share/dotnet/sdk]
  1.1.11 [/usr/share/dotnet/sdk]
  1.1.12 [/usr/share/dotnet/sdk]
  1.1.13 [/usr/share/dotnet/sdk]
  2.0.0 [/usr/share/dotnet/sdk]
  2.0.3 [/usr/share/dotnet/sdk]
  2.1.2 [/usr/share/dotnet/sdk]
  2.1.4 [/usr/share/dotnet/sdk]
  2.1.100 [/usr/share/dotnet/sdk]
  2.1.101 [/usr/share/dotnet/sdk]
  2.1.102 [/usr/share/dotnet/sdk]
  2.1.103 [/usr/share/dotnet/sdk]
  2.1.104 [/usr/share/dotnet/sdk]
  2.1.105 [/usr/share/dotnet/sdk]
  2.1.200 [/usr/share/dotnet/sdk]
  2.1.201 [/usr/share/dotnet/sdk]
  2.1.202 [/usr/share/dotnet/sdk]
  2.1.300 [/usr/share/dotnet/sdk]
  2.1.301 [/usr/share/dotnet/sdk]
  2.1.302 [/usr/share/dotnet/sdk]
  2.1.400 [/usr/share/dotnet/sdk]
  2.1.401 [/usr/share/dotnet/sdk]
  2.1.402 [/usr/share/dotnet/sdk]
  2.1.403 [/usr/share/dotnet/sdk]
  2.1.500 [/usr/share/dotnet/sdk]
  2.1.502 [/usr/share/dotnet/sdk]
  2.1.503 [/usr/share/dotnet/sdk]
  2.1.504 [/usr/share/dotnet/sdk]
  2.1.505 [/usr/share/dotnet/sdk]
  2.1.700 [/usr/share/dotnet/sdk]
  2.2.100 [/usr/share/dotnet/sdk]
  2.2.101 [/usr/share/dotnet/sdk]
  2.2.102 [/usr/share/dotnet/sdk]
  2.2.103 [/usr/share/dotnet/sdk]
  2.2.104 [/usr/share/dotnet/sdk]
  2.2.105 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.5 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.8 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.9 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.11 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.5 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.8 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.9 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.11 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.9 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.10 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.11 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.12 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.13 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.14 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.15 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.2 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.4 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.9 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.10 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.11 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.12 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.9 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.11 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.2 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

+ mono --version
Mono JIT compiler version 5.20.1.19 (tarball Thu Apr 11 09:09:21 UTC 2019)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	Interpreter:   yes
	LLVM:          yes(600)
	Suspend:       hybrid
	GC:            sgen (concurrent by default)
A selection of the failures:

Note that the --disable-parallel is not necessary to reproduce the issue.

Issues with System.Collections.Concurrent:
Command: dotnet restore --verbosity Detailed --disable-parallel

         Paket failed with
         -> DirectoryNotFoundException: Could not find a part of the path "/home/vsts/work/1/s/packages/System.Collections.Concurrent/system.collections.concurrent.4.3.0.nupkg".
   4:5>/home/vsts/work/1/s/.paket/Paket.Restore.targets(138,5): error MSB3073: The command "mono --runtime=v4.0.30319 "/home/vsts/work/1/s/.paket/paket.exe" restore" exited with code 1. [/home/vsts/work/1/s/MainLib/MainLib.fsproj]
       Done executing task "Exec" -- FAILED.
   4:5>Done building target "PaketRestore" in project "MainLib.fsproj" -- FAILED.

(full log here)

         Paket failed with
         -> Error during extraction of /home/vsts/work/1/s/packages/System.Collections.Concurrent/system.collections.concurrent.4.3.0.nupkg.
            
             In rare cases a firewall might have blocked the download. Please look into the file and see if it contains text with further information.
         -> DirectoryNotFoundException: Could not find a part of the path "/home/vsts/work/1/s/packages/System.Collections.Concurrent/lib/portable-net45%2Bwin8%2Bwpa81/_._".
   2:4>/home/vsts/work/1/s/.paket/Paket.Restore.targets(138,5): error MSB3073: The command "mono --runtime=v4.0.30319 "/home/vsts/work/1/s/.paket/paket.exe" restore" exited with code 1. [/home/vsts/work/1/s/MainProj/MainProj.fsproj]
   2:4>Done executing task "Exec" -- FAILED.

(full log here)

Segfaults:
Command: dotnet restore MainProj/MainProj.fsproj --verbosity Detailed --disable-parallel
Seemingly, specifying a project file causes the segfault consistently.

   3:4>PaketRestore:
         mono --runtime=v4.0.30319 "/home/vsts/work/1/s/.paket/paket.exe" restore
         Paket version 5.215.0
         
         =================================================================
         	Native Crash Reporting
         =================================================================
         Got a SIGSEGV while executing native code. This usually indicates
         a fatal error in the mono runtime or one of the native libraries 
         /proc/self/maps:
         00400000-00849000 r-xp 00000000 08:01 35932                              /usr/bin/mono-sgen
         00a49000-00a50000 r--p 00449000 08:01 35932                              /usr/bin/mono-sgen
         00a50000-00a55000 rw-p 00450000 08:01 35932                              /usr/bin/mono-sgen
         00a55000-00a6c000 rw-p 00000000 00:00 0 
         02555000-027bc000 rw-p 00000000 00:00 0                                  [heap]

<snipped for brevity; see full log>
         =================================================================
         	Managed Stacktrace:
         =================================================================
         	  at <unknown> <0xffffffff>
         	  at System.Object:__icall_wrapper_mono_generic_class_init <0x00061>
         	  at Paket.Dependencies:Restore <0x000cf>
         	  at Paket.Dependencies:Restore <0x00063>
         	  at main@889-2:Invoke <0x0003f>
         	  at Paket.Program:processWithValidationEx$cont@42 <0x0009d>
         	  at Paket.Program:processWithValidationEx <0x001c3>
         	  at Paket.Program:main <0x003eb>
         	  at <StartupCode$paket>.$Paket.Program:main@ <0x00037>
         	  at System.Object:runtime_invoke_void <0x00072>
         =================================================================

(full log here)

         =================================================================
         	Managed Stacktrace:
         =================================================================
         	  at <unknown> <0xffffffff>
         	  at Paket.FrameworkDetection:f@62-2 <0x01acf>
         	  at Extract@690-34:Invoke <0x0002b>
         	  at System.Collections.Concurrent.ConcurrentDictionary`2:GetOrAdd <0x000bd>
         	  at Extract@690-33:Invoke <0x0011a>
         	  at Microsoft.FSharp.Collections.ArrayModule:Choose <0x00106>
         	  at Paket.PlatformMatching:f@62-5 <0x0017f>
         	  at extractPlatformsPriv@29-2:Invoke <0x0002b>
         	  at System.Collections.Concurrent.ConcurrentDictionary`2:GetOrAdd <0x000bd>
         	  at extractPlatformsPriv@29-1:Invoke <0x0011a>
         	  at Paket.PlatformMatching:extractPlatforms <0x0010f>
         	  at Paket.Requirements:parseOperator$cont@791 <0x0022f>
         	  at Paket.Requirements:parseOperand@828 <0x000eb>
         	  at Paket.Requirements:parseOperator@768 <0x004a7>
         	  at Paket.Requirements:parseOperand@828 <0x000eb>
         	  at Paket.Requirements:parseOperator@768 <0x004a7>
         	  at Paket.Requirements:parseRestrictionsRaw <0x0007b>
         	  at parseRestrictionsSimplified@880:Invoke <0x0002b>
         	  at InstallSettings:Parse <0x0036a>
         	  at Paket.LockFileParser:|Remote|NugetPackage|NugetDependency|SourceFile|RepositoryType|Group|InstallOption|$cont@368-6 <0x0022f>
         	  at Paket.LockFileParser:|Remote|NugetPackage|NugetDependency|SourceFile|RepositoryType|Group|InstallOption|$cont@285-5 <0x0006f>
         	  at Paket.LockFileParser:Parse$cont@456-2 <0x0004b>
         	  at Microsoft.FSharp.Collections.SeqModule:Fold <0x00117>
         	  at Paket.LockFile:Parse <0x00057>
         	  at lockFile@652:Invoke <0x00057>
         	  at System.Lazy`1:ViaFactory <0x0007d>
         	  at System.Lazy`1:ExecutionAndPublication <0x000d0>
         	  at System.Lazy`1:CreateValue <0x000dd>
         	  at System.Lazy`1:get_Value <0x00031>
         	  at Paket.RestoreProcess:Restore$cont@692 <0x0007a>
         	  at Paket.RestoreProcess:Restore <0x00723>
         	  at Paket.Dependencies:Restore <0x00063>
         	  at main@889-2:Invoke <0x0003f>
         	  at Paket.Program:processWithValidationEx$cont@42 <0x0009d>
         	  at Paket.Program:processWithValidationEx <0x001c3>
         	  at Paket.Program:main <0x003eb>
         	  at <StartupCode$paket>.$Paket.Program:main@ <0x00037>
         	  at System.Object:runtime_invoke_void <0x00072>
         =================================================================

(full log here)

Repro steps

This git repo should work to reproduce the issue: https://github.com/bfrengley/paket-segfault

  1. Fork the above repo
  2. Create an Azure pipeline using the classic editor (not YAML)
    1. Use your copy of the above repo
    2. Select the ASP.NET Core template (not ASP.NET Core (.NET Framework))
    3. Make sure the agent pool is Hosted Ubuntu 1604
    4. Set "Project(s) to restore and build" to **/MainProj.fsproj
  3. Save and queue the build
Expected behavior

dotnet restore should run successfully

Actual behavior

dotnet restore segfaults or otherwise fails

Known workarounds

None.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the restore on Hosted Ubuntu 1604 using the listed .NET SDK and Mono versions, comparing project-file and non-project restores. Inspect .paket/Paket.Restore.targets and the Paket.Dependencies:Restore stack shown in the crash log. Done means restore completes without missing package paths, extraction errors, or a Mono segfault.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, fsharp
Domain
build-system, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.