Difficulties with swipe on container with inner and outer click events

So here is my use case I got some list items that expand on click and they wrap another button. The item itself is swipeable (swiped to left should delete the item). I got this to work but I recognized that swiping also triggers the on click event of my item (case 1 in my sandbox). After some research I thought that onTap could solve this issue. Which it did but it also causes another issue: the inner click also trigger the onTap event even when I stop the propagation of the onClick. This makes sense but I am having a hard time finding a solution for this use case which is not very exotic in my opinion.
Here is the sandbox I am thankful for every answer :)
https://codesandbox.io/p/sandbox/react-swipeable-image-carousel-forked-xv4xjk?file=%2Fsrc%2FInnerClickTriggersTap.tsx%3A21%2C9