decoder was much slower than encoder

here is test log.

```
scguo@scguo-vm:~/Documents/audios$ ../lyra/bazel-bin/encoder_main --model_path=../lyra/wavegru --output_dir=./ --input_path=./12s.wav
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20210422 15:47:21.758337 22733 encoder_main_lib.cc:94] Elapsed seconds : 0
I20210422 15:47:21.758489 22733 encoder_main_lib.cc:95] Samples per second : 9.08575e+06
scguo@scguo-vm:~/Documents/audios$ ../lyra/bazel-bin/decoder_main --model_path=../lyra/wavegru --output_dir=./ --encoded_path=./12s.lyra
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20210422 15:47:34.122872 22742 lyra_wavegru.h:80] lyra_wavegru running in slow generic mode.
I20210422 15:47:34.124763 22742 layer_wrapper.h:96] |lyra_16khz_ar_to_gates_| layer: Shape: [3072, 4]. Sparsity: 0
I20210422 15:47:34.228018 22742 layer_wrapper.h:96] |lyra_16khz_gru_layer_| layer: Shape: [3072, 1024]. Sparsity: 0.9375
I20210422 15:47:34.241358 22742 lyra_wavegru.h:226] Model size: 1271266 bytes
I20210422 15:47:34.241510 22742 wavegru_model_impl.cc:87] Feature size: 160
I20210422 15:47:34.241612 22742 wavegru_model_impl.cc:88] Number of samples per hop: 640
I20210422 15:47:34.246847 22742 layer_wrapper.h:96] |lyra_16khz_conv1d_| layer: Shape: [512, 480]. Sparsity: 0.919987
I20210422 15:47:34.257881 22742 layer_wrapper.h:96] |lyra_16khz_conditioning_stack_0_| layer: Shape: [512, 1024]. Sparsity: 0.920013
I20210422 15:47:34.268566 22742 layer_wrapper.h:96] |lyra_16khz_conditioning_stack_1_| layer: Shape: [512, 1024]. Sparsity: 0.920013
I20210422 15:47:34.279476 22742 layer_wrapper.h:96] |lyra_16khz_conditioning_stack_2_| layer: Shape: [512, 1024]. Sparsity: 0.920013
I20210422 15:47:34.289640 22742 layer_wrapper.h:96] |lyra_16khz_transpose_0_| layer: Shape: [1024, 512]. Sparsity: 0.920013
I20210422 15:47:34.300124 22742 layer_wrapper.h:96] |lyra_16khz_transpose_1_| layer: Shape: [1024, 512]. Sparsity: 0.920013
I20210422 15:47:34.310822 22742 layer_wrapper.h:96] |lyra_16khz_transpose_2_| layer: Shape: [1024, 512]. Sparsity: 0.920013
I20210422 15:47:34.320670 22742 layer_wrapper.h:96] |lyra_16khz_conv_cond_| layer: Shape: [1024, 512]. Sparsity: 0.920013
I20210422 15:47:34.389773 22742 layer_wrapper.h:96] |lyra_16khz_conv_to_gates_| layer: Shape: [3072, 1024]. Sparsity: 0.919998
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20210422 15:47:34.394865 22742 kernels_generic.h:241] SumVectors: using generic kernel!
I20210422 15:47:40.876049 22742 decoder_main_lib.cc:96] Elapsed seconds : 6
I20210422 15:47:40.876096 22742 decoder_main_lib.cc:97] Samples per second : 30309
```
encoder only takes 0 second, but decode takes 6 second.