EnzymeAD / EnzymeAD/Enzyme

Enzyme introduces additional runtime overhead on function calls and for loops

Open
#377 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
LLVM
Stars
1.7k
Forks
188
Avg merge
2d 4h
Merged PRs (30d)
22

Description

Hi,

I've been using Enzyme to differentiate some of my existing c++ code. I found that when there are for loops or function calls existing inside the function, the reverse pass would be very slow. Then I tried to move the for loop outside the function being differentiated, the reverse pass would run 2x faster. I made a simple example below to show this problem, In `d_sum2`, the for loop is inside `__enzyme_autodiff`, while in `d_sum1`, the for loop is outside `__enzyme_autodiff`. `d_sum1` ran 2x faster than `d_sum2'. I'm not sure if I'm using Enzyme incorrectly or Enzyme does introduce some additional overhead. Thank you.

``` cpp
#include
#include "timer.h"
#include
#include
#include
using namespace std;
void __enzyme_autodiff(...);

int enzyme_dup, enzyme_const, enzyme_out, enzyme_dupnoneed;

__attribute__((optnone)) double sum1(std::vector &v)
{
double ret = 0.;
for (int i = 0; i < v.size(); i++)
{
ret += v[i];
}
ret /= v.size();
return ret;
}

__attribute__((optnone)) double sum2(std::vector &v)
{
double res = 0.;
for (int i = 0; i < v.size(); i++)
{
res += sum1(v);
}
return res;
}

__attribute__((optnone)) void d_sum2(std::vector &v, std::vector &d_v)
{
__enzyme_autodiff((void *)(sum2),
enzyme_dup, &v, &d_v);
}

__attribute__((optnone)) void d_sum1(std::vector &v, std::vector &d_v)
{
double res = 1.;
for (int i = 0; i < v.size(); i++)
{
__enzyme_autodiff((void *)(sum1),
enzyme_dup, &v, &d_v);
}
}

int main()
{
std::vector v(1000);
std::vector dv(1000);
int size = 100;
for (auto &e : v)
e = 1.;
for (auto &e : dv)
e = 0.;
{
Timer timer("s");
for (int i = 0; i < 100; i++)
d_sum1(v, dv);
}
printf("%f\n", dv[0]);

{
Timer timer("s");
for (int i = 0; i < 100; i++)
d_sum2(v, dv);
}
printf("%f\n", dv[0]);
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.