Investigate how we can load external assemblies when running Lucene.Net.Benchmarks on the command line
- Dominant language
- C#
- Stars
- 2.4k
- Forks
- 658
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 9
Description
The benchmarks project was designed to be able to load user-defined projects to run. In Java, this could be done with a single string to identify the types to load, however, .NET requires a reference to the actual assembly in order to read the types from it.
We currently have it set up to read all types from all assemblies that are referenced, but this causes the `Lucene.Net.Tests.Benchmark.ByTask.Tasks.Alt::TestWithoutAlt()` test to fail because in Java the types were supposed to be loaded on demand. So, we need to investigate the best way to load types from external assemblies in .NET to run benchmarks on from the `lucene-cli` tool.
The [part that has been altered](https://github.com/apache/lucenenet/blob/f0930a5f8df377cccc428dcfeba02ea4098a6dee/src/Lucene.Net.Benchmark/ByTask/Utils/Algorithm.cs#L359-L385) to allow assemblies to be "automatically" discovered is:
```c#
// Loads all assemblies in current referenced project (this was not in the original Lucene source)
IEnumerable referencedAssemblies = AssemblyUtils.GetReferencedAssemblies().Select(a => a.GetName().Name);
result.Add(dfltPkg);
if (alts == null)
{
result.UnionWith(referencedAssemblies);
return result.ToArray();
}
foreach (string alt in alts.Split(',').TrimEnd())
{
result.Add(alt);
}
result.UnionWith(referencedAssemblies);
```
[Equivalent in Lucene 4.8.1](https://github.com/apache/lucene-solr/blob/f01152a5909fa6059f4f1d4aeb4e14968ef1d8c2/lucene/benchmark/src/java/org/apache/lucene/benchmark/byTask/utils/Algorithm.java#L290-L302)
The problem is that when running as a separate process (`lucene-cli`), the end user has no way to reference assemblies, and therefore cannot change what is loaded by the tool.
I am no expert on Java, but from what I gather there is a convention-based and extensible "class path" that can be interacted with by end users regardless of whether it is inside or outside of the `.jar` package. I think the way it works is that by simply dropping an external `.class` file (similar to a .NET `Type`) in the same directory as an internal class, the JVM will load it, but it is also possible to inject a custom "class loader" to load from alternate locations or to add additional class paths on the command line.
It is also possible in Java to either reference a `.jar` file like a DLL or execute it like an EXE. For example, if a `main()` method exists in any class, it can be executed directly on the `.jar` file like:
```
java -cp lucene-core.jar org.apache.lucene.index.IndexUpgrader [-delete-prior-commits] [-verbose] indexDir
or
java -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex pathToIndex [-fix] [-verbose] [-segment X] [-segment Y]
```
Since DLLs cannot be executed directly in .NET in the same way, we have a gap between the two platforms. The `lucene-cli` tool was created as a wrapper process to execute any one of the `main()` methods that were included in Lucene to fill this gap. This wrapper executable has revealed yet another gap, since the end user should be able to supply their own classes to benchmark and currently there is no way to do so.
So, the essence of this task is to do the following:
- Create a similar convention-based and/or command-line based way for end users to be able to run any of the [benchmark commands](https://github.com/apache/lucenenet/tree/f0930a5f8df377cccc428dcfeba02ea4098a6dee/src/dotnet/tools/lucene-cli/docs/benchmark) against their own assemblies/code
- Prefer to utilize the .NET platform's nearest counterpart and convention rather than invent a custom one, where possible
- Utilize the Lucene.Net.Benchmarks string-based configuration for the user to be able to specify which type to load
IMO, we don't necessarily have to have as many options as were available in Java, we simply need to provide *an* option, which we are currently lacking.
Contributor guide
Research direction
Start with the altered assembly-discovery code in src/Lucene.Net.Benchmark/ByTask/Utils/Algorithm.cs around lines 359-385, the failing Lucene.Net.Tests.Benchmark.ByTask.Tasks.Alt::TestWithoutAlt() test, and the lucene-cli benchmark documentation. Investigate the .NET convention or command-line mechanism for supplying external assemblies while preserving the string-based type configuration. Done means users can run benchmark commands against their own assemblies and the existing test no longer fails.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- cli, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100