Random builds throw ReflectionTypeLoadException exception for same codebase (no changes)
- Dominant language
- PowerShell
- Stars
- 22k
- Forks
- 4.9k
- Avg merge
- 5d 5h
- Merged PRs (30d)
- 29
Description
### Description
During random builds generated through GitHub actions on Amazon EKS, upon running these specific builds always get `ReflectionTypeLoadException` thrown saying - `Could not load file or assembly 'SomeAssemblyName, Version=VersionNumber, Culture=neutral, PublicKeyToken=SomeToken'. The system cannot find the file specified.`
### Project Details
1. It is a multi-project based solution (API, Infra, Core, etc. )
2. API project is the start-up project
3. The solution is based upon `dotnet 8`
4. We use GitHub actions with self hosted runner to build and deploy the project and later run it on lambda.
### Necessary files and commands used
- API project's .csproj
```csproj
Exe
net8.0
enable
enable
true
Lambda
true
false
Always
```
- dotnet CLI commands used in GitHub pipeline to publish project/artifact
```bash
dotnet restore
dotnet publish -c Release -r linux-x64 --self-contained false -o ./publish
```
### Expected Behaviour
When running the code via dotnet cli, the published artifact should run without any errors for every build created using same codebase on lambda (given that there are no other errors in codebase).
### Actual Behaviour
When lambda tries to run the published artifacts, some artifacts (that were generated via same codebase) always throw `ReflectionTypeLoadException` exception. Only after couple of attempts to re-publishing same code, this issue gets resolved on its own.
I get the same error as the lambda error, if I try running the GitHub artifact on my local windows machine using `dotnet platform_jjz_service_api.dll`
Stack trace for runtime exception
```csharp
Unhandled exception. System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5'. The system cannot find the file specified.
at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
at MediatR.Registration.ServiceRegistrar.<>c.b__1_0(Assembly a)
at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
at MediatR.Registration.ServiceRegistrar.ConnectImplementationsToTypesClosing(Type openRequestInterface, IServiceCollection services, IEnumerable`1 assembliesToScan, Boolean addIfAlreadyExists, MediatRServiceConfiguration configuration)
at MediatR.Registration.ServiceRegistrar.AddMediatRClasses(IServiceCollection services, IEnumerable`1 assembliesToScan, MediatRServiceConfiguration configuration)
at MediatR.ServiceCollectionExtensions.AddMediatR(IServiceCollection services, IEnumerable`1 assemblies, Action`1 configuration)
at MediatR.ServiceCollectionExtensions.AddMediatR(IServiceCollection services, Assembly[] assemblies)
at platform_jjz_service_api.Startup.ConfigureServices(IServiceCollection services) in /runner/_work/clin/clin/platform_jjz_service_api/Startup.cs:line 292
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services, Object instance)
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass7_0.b__0(HostBuilderContext context, IServiceCollection services)
at Microsoft.Extensions.Hosting.HostBuilder.InitializeServiceProvider()
at Microsoft.Extensions.Hosting.HostBuilder.Build()
at platform_jjz_service_api.LocalEntryPoint.Main(String[] args) in /runner/_work/clin/clin/platform_jjz_service_api/LocalEntryPoint.cs:line 10
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5'. The system cannot find the file specified.
File name: 'Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5'
```
### My findings
1. This is a non-deterministic error, which randomly occurs (only some builds throw this runtime exception, other builds for same code work totally fine)
2. When compared an erroneous artifact with a correct artifact, noticed that number of libraries in `platform_jjz_service_api.deps.json` is lesser in erroneous artifact.
3. Changed the dotnet commands slightly to publish the release artifact, but no luck there :
```
dotnet nuget locals all --clear
dotnet restore --locked-mode
dotnet publish -c Release -r linux-x64 --self-contained false -o ./publish
```
### Known Workarounds
Whenever this issue occurs, just keep re-deploying the same code multiple times, until this issue goes away in one of the fresher builds.
Contributor guide
Assessment
This issue has not been assessed yet.