Button Click not detected

I will try to explain this issue as clearly as I can. In my layoutview I have a button that the user can click to open the slider to show more buttons below it. but some times the button clicks aren't being detected unless i press the button multiple times.

For example if I have 3 rows and I press 1st, 2nd, 3rd row it all opens correctly. But after the 3rd row if I press the 1st row it doesn't detect the click and requires multiple touches / clicks. However if i got 1st, 2nd, 3rd row then back to 2, the 2nd row opens just fine. not really sure why this weird behaviour is happening. Here is my XML - Button id is overflow

`
< com.daimajia.swipe.SwipeLayout
xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:id="@+id/swipe"
app:show_mode="pull_out">

```

`
```

If I remove the Swipe layout I can see that the button click is detected but once I put it back in it does seem to not respond at times. Can some one please help