Unity Mono – A Very Simple Benchmark ~2016-06-28

When I say the Mono version that ships with Unity is slow, people get curious. I’d say it’s substantially slower compared to the recent versions of the MS .NET Framework. This is a quick post so I’m not going into the details of individual numbers for the CLR overhead.

If you’re not familiarized with JIT-compiled languages, each one has its own peculiarities. In case of C#, along with other .NET languages, it’s compiled into an intermediary language called IL. The IL code is JIT-compiled at execution time by a “virtual machine” (CLR in case of .NET) which technically can be considered an interpreter of kinds (if that helps you wrap your head around it) that outputs native bytecode. This happens on-the-fly on a per-routine basis (in most cases), and can cause the infamous “JIT spikes” the first time you call a function – but there are ways to “pre-JIT” your functions beforehand to prevent that. The advocates for high-performance JIT-compiled languages promise they are capable of outputting native code that’s faster than ahead-of-time (AOT) compiled binaries because the JIT compiler can ‘tune’ its output specifically for each architecture, unfortunately in real-life applications they’re rarely faster than hand-optimized C++ code, although an example I’ve seen of JIT being faster is the Farseer physics engine, but it’s heavily hand-optimized. It makes sense that processor specifics intrinsics (SIMD) get compiled appropriatedly in a JIT fashion. Another solution that in my opinion is a better option is install-time compilation.

I’ve benchmarked addition, subtraction, multiplication, division and modulo operations with uints+uints and floats+uints types (all 32 bit). Here are the results (seconds, Unity 5.3.5f1 and .NET 4.6.1 on my testing rig):

Analyzing that graph we notice that both lines are more or less parallel until the last 2 results. Unity took about 5 extra seconds to compute each of these tests. In the last 2 tests we got mixed results, but I was expecting those operations to be much slower than the other ones. If we had a literal value in the operations there, for example x/2, the IL compiler probably could optimize it to use multiplication instead, but since it’s a variable that’s not possible, so MultFloat costing the same as DivFloat in Mono is a mystery to me.

Perhaps the explanation is that Mono is ignoring the attributes NOT to optimize the functions, and applying its IL optimization to it anyway, that is an especially valid hypothesis when you allow the IL compiler to optimize the code, in which case the Mono numbers are exactly the same, but the numbers from MS .NET are very different:
In the last result Mono is 24 times slower, however, the last 5 results shouldn’t be used for comparison without inspecting the IL code. Please note this is 100% automated, I just commented out the “MethodImpl” attributes in the code below… Without a careful analysis I can only make conjectures: maybe the IL compiler was smart enough to discard the operations altogether; maybe the JIT compiler is using some extended instruction set (very unlikely to make that difference).

In the future I’m going to do more extensive tests (collections and whatnot) and look at the IL code generated by both compilers to compare them. I think this synthetic benchmark doesn’t reflect the reality very well (well, they never do), because I feel that Unity is much slower than that in real-life, or maybe I’m just biased… the numbers never lie though. Something to consider is that the benchmark doesn’t generate garbage, and the garbage collection in Unity is probably what is lagging behind the most.

I know that Unity has a very old Mono version because of licensing issues with Xamarin. Now that Microsoft bought Xamarin, I hope that changes quickly. In any case, it would be nice if Unity could use the MS .NET Framework on Windows platforms because it’s even faster than the latest Mono versions I’ve tried. But perhaps I’m asking too much.

Here’s the fairly simple code:
```
class Program
{
const uint Iterations = UInt32.MaxValue/8;
static void Main(string\[\] args)
{

GC.Collect();
GC.WaitForPendingFinalizers();
_bogus = 0;

Console.WriteLine("Benchmark Iterations: "+Iterations);
Measure(SumInt);
Measure(SubInt);
Measure(MultInt);
Measure(DivInt);
Measure(ModInt);
Measure(SumFloat);
Measure(SubFloat);
Measure(MultFloat);
Measure(DivFloat);
Measure(ModFloat);
//Console.WriteLine("Benchmarks Finished");Console.ReadKey();
}

\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void Measure(Action function)
{
Stopwatch sw = Stopwatch.StartNew();
function();
sw.Stop();
Console.WriteLine(function.Method.Name+" took:\\t"+sw.Elapsed);
}

private static long _bogus;
\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void Bogus(float foo) { _bogus+=foo; }
\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void Bogus(uint foo) { _bogus+=foo; }

//BENCHMARK FUNCTIONS

\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void SumInt()
{
uint c = 0;
for (uint i = 1; i < Iterations; i++)
{
Bogus(c + i);
}
}

\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void SubInt()
{
uint c = Iterations;
for (uint i = 1; i < Iterations; i++)
{
Bogus(c - i);
}
}

\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void MultInt()
{
uint c = 2;
for (uint i = 1; i < Iterations; i++)
{
Bogus(c * i);
}
}

\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void DivInt()
{
uint c = Iterations;
for (uint i = 1; i < Iterations; i++)
{
Bogus(c / i);
}
}

\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void ModInt()
{
uint c = Iterations;
for (uint i = 1; i < Iterations; i++)
{
Bogus(c % i);
}
}

\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void SumFloat()
{
float c = 0;
for (uint i = 1; i < Iterations; i++)
{
Bogus(c + i);
}
}

\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void SubFloat()
{
float c = Iterations;
for (uint i = 1; i < Iterations; i++)
{
Bogus(c - i);
}
}

\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void MultFloat()
{
float c = 2;
for (uint i = 1; i < Iterations; i++)
{
Bogus(c * i);
}
}

\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void DivFloat()
{
float c = Iterations;
for (uint i = 1; i < Iterations; i++)
{
Bogus(c / i);
}
}

\[MethodImpl(MethodImplOptions.NoInlining | MethodImplOptions.NoOptimization)\]
static void ModFloat()
{
float c = Iterations;
for (uint i = 1; i < Iterations; i++)
{
Bogus(c % i);
}
}

}
```
For the Unity version, besides using an empty scene not even with the default sky, I’ve added some extra code to ensure a fair comparison:
```
public class Benchmarks : MonoBehaviour {

void Start ()
{
StartCoroutine(WaitAndBench());
}

IEnumerator WaitAndBench()
{
yield return new WaitForSeconds(1);
Bench();
}

const uint Iterations = UInt32.MaxValue / 8;
static void Bench()
{
//\[... MAIN CODE GOES HERE AND THE REST IS THE SAME ...\]
```
Author [alan](http://alangamedev.com/author/alan/)Posted on [June 28, 2016June 25, 2017](http://alangamedev.com/unity-mono-a-very-simple-benchmark/)Categories [Benchmarks](http://alangamedev.com/category/benchmarks/), [Unity](http://alangamedev.com/category/unity/) [Edit "Unity Mono – A Very Simple Benchmark"](http://alangamedev.com/wp-admin/post.php?post=64&action=edit)