apache / apache/lucenenet

Investigate how we can load external assemblies when running Lucene.Net.Benchmarks on the command line

Open
#305 6 comments 0 reactions 0 assignees View on GitHub
benchmarks investigation is:enhancement pri:normal up-for-grabs
Dominant language
C#
Stars
2.4k
Forks
658
Avg merge
3d 5h
Merged PRs (30d)
9

Description

The benchmarks project was designed to be able to load user-defined projects to run. In Java, this could be done with a single string to identify the types to load, however, .NET requires a reference to the actual assembly in order to read the types from it.

We currently have it set up to read all types from all assemblies that are referenced, but this causes the `Lucene.Net.Tests.Benchmark.ByTask.Tasks.Alt::TestWithoutAlt()` test to fail because in Java the types were supposed to be loaded on demand. So, we need to investigate the best way to load types from external assemblies in .NET to run benchmarks on from the `lucene-cli` tool.

The [part that has been altered](https://github.com/apache/lucenenet/blob/f0930a5f8df377cccc428dcfeba02ea4098a6dee/src/Lucene.Net.Benchmark/ByTask/Utils/Algorithm.cs#L359-L385) to allow assemblies to be "automatically" discovered is:

```c#
// Loads all assemblies in current referenced project (this was not in the original Lucene source)
IEnumerable referencedAssemblies = AssemblyUtils.GetReferencedAssemblies().Select(a => a.GetName().Name);
result.Add(dfltPkg);

if (alts == null)
{
result.UnionWith(referencedAssemblies);
return result.ToArray();
}

foreach (string alt in alts.Split(',').TrimEnd())
{
result.Add(alt);
}
result.UnionWith(referencedAssemblies);
```

[Equivalent in Lucene 4.8.1](https://github.com/apache/lucene-solr/blob/f01152a5909fa6059f4f1d4aeb4e14968ef1d8c2/lucene/benchmark/src/java/org/apache/lucene/benchmark/byTask/utils/Algorithm.java#L290-L302)

The problem is that when running as a separate process (`lucene-cli`), the end user has no way to reference assemblies, and therefore cannot change what is loaded by the tool.

I am no expert on Java, but from what I gather there is a convention-based and extensible "class path" that can be interacted with by end users regardless of whether it is inside or outside of the `.jar` package. I think the way it works is that by simply dropping an external `.class` file (similar to a .NET `Type`) in the same directory as an internal class, the JVM will load it, but it is also possible to inject a custom "class loader" to load from alternate locations or to add additional class paths on the command line.

It is also possible in Java to either reference a `.jar` file like a DLL or execute it like an EXE. For example, if a `main()` method exists in any class, it can be executed directly on the `.jar` file like:

```
java -cp lucene-core.jar org.apache.lucene.index.IndexUpgrader [-delete-prior-commits] [-verbose] indexDir

or

java -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex pathToIndex [-fix] [-verbose] [-segment X] [-segment Y]
```

Since DLLs cannot be executed directly in .NET in the same way, we have a gap between the two platforms. The `lucene-cli` tool was created as a wrapper process to execute any one of the `main()` methods that were included in Lucene to fill this gap. This wrapper executable has revealed yet another gap, since the end user should be able to supply their own classes to benchmark and currently there is no way to do so.

So, the essence of this task is to do the following:

- Create a similar convention-based and/or command-line based way for end users to be able to run any of the [benchmark commands](https://github.com/apache/lucenenet/tree/f0930a5f8df377cccc428dcfeba02ea4098a6dee/src/dotnet/tools/lucene-cli/docs/benchmark) against their own assemblies/code
- Prefer to utilize the .NET platform's nearest counterpart and convention rather than invent a custom one, where possible
- Utilize the Lucene.Net.Benchmarks string-based configuration for the user to be able to specify which type to load

IMO, we don't necessarily have to have as many options as were available in Java, we simply need to provide *an* option, which we are currently lacking.

Contributor guide

Open the contributing guide

Research direction

Start with the altered assembly-discovery code in src/Lucene.Net.Benchmark/ByTask/Utils/Algorithm.cs around lines 359-385, the failing Lucene.Net.Tests.Benchmark.ByTask.Tasks.Alt::TestWithoutAlt() test, and the lucene-cli benchmark documentation. Investigate the .NET convention or command-line mechanism for supplying external assemblies while preserving the string-based type configuration. Done means users can run benchmark commands against their own assemblies and the existing test no longer fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
cli, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.