v3 beta fail to pass server response skb to client

I haven't got v3 beta release working so far, the syncookie issue disappeared, but it always seems that LVS failed to pass the response from server to client, the debugging log as below with my added syncookie debug noise:

Jan 10 20:43:46 alilvs kernel: [ 6379.609084] device eth0 entered promiscuous mode
Jan 10 20:43:56 alilvs kernel: [ 6389.510018] device eth1 entered promiscuous mode
Jan 10 20:44:02 alilvs kernel: [ 6395.506678] syn_cookie check cookie=81872209
Jan 10 20:44:02 alilvs kernel: [ 6395.506681] syn_cookie check vs syn seq=2095846056
Jan 10 20:44:02 alilvs kernel: [ 6395.506684] syn_cookie check count=71684
Jan 10 20:44:02 alilvs kernel: [ 6395.506686] syn_cookie check diff=0
Jan 10 20:44:02 alilvs kernel: [ 6395.506687] syn_cookie check excuted second
hash
Jan 10 20:44:02 alilvs kernel: [ 6395.506690] syn_cookie check res=2125824
Jan 10 20:44:02 alilvs kernel: [ 6395.506692] syn_cookie check mssind=7
Jan 10 20:44:02 alilvs kernel: [ 6395.506694] syn_cookie check NUM_MSS=10
Jan 10 20:44:02 alilvs kernel: [ 6395.506697] IPVS: ip_vs_rr_schedule(): Scheduling...
Jan 10 20:44:02 alilvs kernel: [ 6395.506701] IPVS: RR: server 192.168.3.3:80 activeconns 0 refcnt 1 weight 1
Jan 10 20:44:02 alilvs kernel: [ 6395.506712] IPVS: Bind-dest TCP c:192.168.1.7:43826 v:192.168.1.169:80 d:192.168.3.3:80 fwd:F s:0 conn->flags:185 conn->refcnt:1 dest->refcnt:2
Jan 10 20:44:02 alilvs kernel: [ 6395.506719] IPVS: Schedule fwd:F c:192.168.1.7:43826 v:192.168.1.169:80 d:192.168.3.3:80 conn->flags:81C5 conn->refcnt:2
Jan 10 20:44:02 alilvs kernel: [ 6395.506726] syn_proxy_send_rs_syn netdevice:eth0
Jan 10 20:44:02 alilvs kernel: [ 6395.506729] IPVS: save_xmit_info, skb->dev is NULL.
Jan 10 20:44:02 alilvs kernel: [ 6395.507263] in syn_proxy_synack_rcv, seq = 1230729923 ack_seq = 1294869472 SA- cp->is_synproxy = 32768 cp->state = 2
Jan 10 20:44:02 alilvs kernel: [ 6395.507267] synproxy_save_fast_xmit netdevice:eth1
Jan 10 20:44:02 alilvs kernel: [ 6395.507270] IPVS: save_xmit_info, netdevice:eth0
Jan 10 20:44:02 alilvs kernel: [ 6395.507273] tcp_dnat_handler: tcph->ack_seq 1312875283 => 1230729924, delta = 82145359
Jan 10 20:44:02 alilvs kernel: [ 6395.507276] IPVS: ip_vs_fast_xmit: send skb to RS!
Jan 10 20:44:02 alilvs kernel: [ 6395.507284] IPVS: save_xmit_info, netdevice:eth0
Jan 10 20:44:02 alilvs kernel: [ 6395.507287] tcp_dnat_handler: tcph->ack_seq 1312875283 => 1230729924, delta = 82145359
Jan 10 20:44:02 alilvs kernel: [ 6395.507289] IPVS: ip_vs_fast_xmit: send skb to RS!
Jan 10 20:44:02 alilvs kernel: [ 6395.507718] in syn_proxy_synack_rcv, seq = 1230729924 ack_seq = 1294869637 -A- cp->is_synproxy = 32768 cp->state = 1
Jan 10 20:44:02 alilvs kernel: [ 6395.507722] IPVS: eth1(): netdevice:ip_vs_save_xmit_inside_info
Jan 10 20:44:02 alilvs kernel: [ 6395.507726] tcp_snat_handler: tcph->seq 1230729924 => 1312875283, delta = 82145359
Jan 10 20:44:02 alilvs kernel: [ 6395.507729] IPVS: ip_vs_fast_response_xmit: send skb to client!
Jan 10 20:44:02 alilvs kernel: [ 6395.508013] IPVS: Unbind-dest TCP c:192.168.1.7:43826 v:192.168.1.169:80 d:192.168.3.3:80 fwd:F s:1 conn->flags:8005 conn->refcnt:1 dest->refcnt:2
Jan 10 20:44:02 alilvs kernel: [ 6395.508021] IPVS: Unbind-laddr TCP c:192.168.1.7:43826 v:192.168.1.169:80 l:192.168.3.168:5015 d:192.168.3.3:80 fwd:F s:1 conn->flags:8005 conn->refcnt:1 local->refcnt:2

I initially thought this might be caused by the ip_vs_fast_response_xmit in v3 release, but even disabling fast_response_xmit has no effect

Start by reproducing the v3 beta failure and tracing the response path shown in the kernel log, especially ip_vs_fast_response_xmit and the SYNPROXY handlers. Compare behavior with fast_response_xmit disabled and determine why the server response does not reach the client. Done means the response is reliably forwarded to the client without breaking the connection.

Written by the indexing model from the issue text.