intel / intel/llvm

[SYCL] Poor Performance - Low bandwidth of SYCL kernel for Layer Normalization

Open
#8,030 14 comments 0 reactions 0 assignees View on GitHub
bug cuda
Dominant language
LLVM
Stars
1.5k
Forks
854
Avg merge
3d 17h
Merged PRs (30d)
137

Description

**Describe the bug**

This reproducer is created for enhancing the performance i.e., achieving higher bandwidth for the SYCL implementation of
Layer Normalization primitive on Nvidia.
This reproducer performs the layer normalization operation and calculates the memory bandwidth utilized by the kernel for forward and backward propagations of the primitive.

**To Reproduce**

1. For the reproducer code, refer the attachments _setup.sh_, _ref_layer_normalizations.cpp_ and _ref_layer_normalizations_kernels.hpp_

2. Go to the directory having the reproducer files and run the below script to setup the environment.
`source setup.sh`

To compile, run.
`clang++ -O2 -fsycl -fsycl-targets=nvptx64-nvidia-cuda ref_layer_normalizations.cpp`

3. The above generates the output file. To see the output bandwidth, run
`./a.out`

To test the code for different inputs, Please refer Additional Context section.

4. The below illustrates the observed and expected behaviour.

##### Observed behavior
For both higher and lower input dimensions of N & C, the bandwidth observed is low against the expected.
For higher values, say N = 24576 and C = 1024 with all flags ENABLED (refer Additional Context section), the bandwidth observed is around 10-15%.

```
------Forward Propagation-----
Bandwidth :24.092681 gb/sec

------Backward Propagation-----
kernel--1--Bandwidth :31.919715 gb/sec
Kernel--2--Bandwidth :12.005970 gb/sec
```

##### Expected behavior

The ideal behavior is to attain the maximum bandwidth (i.e., CLPeak value for Float16 = 251.94 GBPS) for any input size.
For the current reproducer at least 40-60% bandwidth is expected i.e., a minimum of 100 GBPS is desired.

**Environment**

- OS: Ubuntu 22.04.1 LTS
- Target device and vendor: Nvidia, Tesla T4
- DPC++ version: clang version 15.0.0 (https://github.com/intel/llvm.git 0c7a1e18978754451f5c2c95129721297e2c2411)
- Dependencies version: Driver Version: 495.29.05 CUDA Version: 11.5

**Additional context**

Note: The inputs for the primitives in this reproducer are hard coded.
Please test all the different combinations of the below inputs and flags for both forward and backward propagations.

The reproducer can be tested by switching the input values from the lines 17 - 26 of ref_layer_normalizations.cpp as illustrated below.
1. N & C values are input dimensions and their values for testing, N ranges from 1 - 24576 and C ranges from 1 - 1024. For Example,
```
N = 4532;
C = 350;
```

2. To enable forward propagation, make `prop_fwd = 1`, in case of backward propagation, `prop_bwd = 1`

3. The below are the layer normalization specific flags which are common for both forward and backward propagations.
Here the value 0 disables the flag and 1 enables the flag. For example,

```
use_scale = 0;
use_shift = 1;
calculate_stats = 1;
save_stats = 0;
```

4. Unlike the forward propagation, The backward propagation contains two kernels - Kernel1 & kernel2. By default, Kernel2 is launched. kernel1 is launched, only when any one of use_scale or use_shift is enabled.

5. Please run _a.out_ multiple times after compilation to see the accurate results.

6. Below contains the code file

[lnorm_reproducer.zip](https://github.com/intel/llvm/files/10444417/lnorm_reproducer.zip)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.