indygreg / indygreg/python-zstandard

Streaming sequentially through open socket not working

Open
#53 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
642
Forks
116
Avg merge
1d 14h
Merged PRs (30d)
5

Description

I am trying to send numpy arrays through an open socket:

server:

import zstandard as zstd
from numpy import *
import json_tricks as json
import socketserver as SocketServer
import sys

x = array([1,2,3,4,5,6,7,8,9])
y = array(['a','b','c','d','e','f','g','h','i','j'])
li = [x,y]

HOST, PORT = "localhost", 9999

class MyTCPSocketHandler(SocketServer.StreamRequestHandler):

def handle(self):

sock = self.wfile
for i in range(2):
val = li[i]
enc = json.dumps(val).encode('utf-8')
cctx = zstd.ZstdCompressor()
with cctx.stream_writer(sock) as compressor:
compressor.write(enc)

HOST, PORT = "localhost", 9999
server = SocketServer.TCPServer((HOST, PORT), MyTCPSocketHandler)
print('ok')
server.serve_forever()

client:

import socket, sys
import numpy
import zstandard as zstd

HOST, PORT = "localhost", 9999

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

try:

sock.connect((HOST,PORT))
sock.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)

mf = sock.makefile()

for i in range(2):
data = ''
dctx = zstd.ZstdDecompressor()
for chunk in dctx.read_to_iter(mf.buffer, read_size=2):
data += chunk.decode('utf-8')

print(data)

finally:
pass

Any read_size specified on the client end that isn't 1 will cause an error:

zstd.ZstdError: zstd decompress error: Unknown frame descriptor.

In this case, the first array is sent and received without problem, but the second (and any more, if there are more) will fail to transmit. In fact, when read_size=1 is specified, the size of the chunk is usually closer to 10kb. I want a chunk size of 1mb but it does not seem to work.

When it comes to sending a larger amount of data, a different error occurs.
Instead of receiving this error message the client will hang while it receives. After closing the connection on the server side (by terminating the program), then sometimes the client will suddenly receive all the data. Is there a flush() call being missed in the zstandard code?

If it helps, everything here works perfectly fine if I replace the zstandard library with [lz4framed](https://github.com/Iotic-Labs/py-lz4framed).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the server/client example using ZstdCompressor.stream_writer and ZstdDecompressor.read_to_iter, first with read_size values of 1 and 2. Investigate the sequential frame and large-transfer behavior over the open socket, then add a regression test covering the expected streamed output and completion behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.