SwipeLayout inside ViewGroup inside SwipeRefreshLayout

SwipeRefreshLayout is the official "pull to refresh" layout. I have a ListView that is inside of a SwipeRefreshLayout, and I am using SwipeLayout as my list item rows.

Is there a good way for me to stop the "pull down" action of the SwipeRefreshLayout when I am horizontally scrolling the SwipeLayout when at the top of the ListView? What happens is that when at the top of the ListView, it is really difficult to horizontally pull the SwipeLayout without accidentally resetting it due to the "pull down" action of the SwipeRefreshLayout activating.