UDP - sockets not duplex in threads

Hi there, I believe I've found a bug with UDP sockets, or at least they don't behave the way I expect.

If I open a TCP connection to a service, I can spawn a reader thread and writer thread on the socket. If I send while the TCP Socket is reading, it works.

With a UDP socket, if I try to send while a UDP socket is reading, I receive a socket busy message.

Here's a few small programs to demonstrate the behavior:

First, a TCP echo server (in Python):

#!/usr/bin/env python3

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

server_address = '0.0.0.0'
server_port = 31337

server = (server_address, server_port)
sock.bind(server)
sock.listen(1)
print("Listening on " + server_address + ":" + str(server_port))

connection, client_address = sock.accept()
print("Connection from " + str(client_address))

while True:
	payload = connection.recv(4096)
	if not payload:
		connection.close()
		break
	print(payload)
	sent = connection.sendall(payload)

And a similar UDP-based echo server, also in Python:

#!/usr/bin/env python3

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

server_address = '0.0.0.0'
server_port = 31337

server = (server_address, server_port)
sock.bind(server)
print("Listening on " + server_address + ":" + str(server_port))

while True:
	payload, client_address = sock.recvfrom(4096)
	print("Echoing data back to " + str(client_address))
	print(payload)
	sent = sock.sendto(payload, client_address)

Here's a client script (Lua) for connecting to TCP, sending packets, and reading responses:

local tcp = ngx.socket.tcp()
tcp:connect('127.0.0.1',31337)

local r = ngx.thread.spawn(function()
  while true do
    local packet, err = tcp:receiveany(4096)
    print(#packet,packet)
  end
end)

local w = ngx.thread.spawn(function()
  local i = 1
  local bytes, err
  while true do
    ngx.sleep(1)
    bytes, err = tcp:send('message ' .. i)
    if err then
      error(err)
    end
    i = i + 1
  end
end)

ngx.thread.wait(r,w)
ngx.thread.kill(r)
ngx.thread.kill(w)

And the equivalent for UDP:

local udp = ngx.socket.udp()
udp:setpeername('127.0.0.1',31337)

local r = ngx.thread.spawn(function()
  while true do
    local packet, err = udp:receive()
    print(#packet,packet)
  end
end)

local w = ngx.thread.spawn(function()
  local i = 1
  local bytes, err
  while true do
    ngx.sleep(1)
    bytes, err = udp:send('message ' .. i)
    if err then
      error(err)
    end
    i = i + 1
  end
end)

ngx.thread.wait(r,w)
ngx.thread.kill(r)
ngx.thread.kill(w)

When I run the TCP version, I get the expected behavior - about once per second, I send a string of text and receive it.

I'd expect the UDP version to behave the same, but it throws the socket busy error.

I'm running both of the Lua scripts with the resty CLI.

I'm on Arch Linux, here's my uname:

Linux hefty 5.10.9-rt24-xanmod1-1-xanmod-rt #1 SMP PREEMPT_RT Thu, 28 Jan 2021 16:44:29 +0000 x86_64 GNU/Linux

I'm using the latest release version of OpenResty (1.19.3.1), here's the output of openresty -V:

nginx version: openresty/1.19.3.1
built by gcc 10.2.0 (GCC) 
built with OpenSSL 1.1.1i  8 Dec 2020
TLS SNI support enabled
configure arguments: --prefix=/home/john/openresty/nginx --with-cc-opt='-O2 -I/home/john/openresty/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.19 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.9 --with-ld-opt='-Wl,-rpath,/home/john/openresty/luajit/lib -s -Wl,-rpath,/home/john/openresty/lib:/home/john/openresty/luajit/lib -L/home/john/openresty/lib' --with-file-aio --with-ipv6 --with-md5-asm --with-pcre-jit --with-sha1-asm --with-threads --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_ssl_module

Start by reproducing the behavior with the Lua ngx.socket.udp and ngx.thread.spawn example under the resty CLI, comparing it with ngx.socket.tcp. Trace the UDP receive and send paths to identify why concurrent operations produce “socket busy.” Done means UDP send and receive can run in separate threads as shown, with a regression test covering the behavior.