Dealing with non-contiguous memoryviews as binary buffers
- Lenguaje dominante
- Python
- Estrellas
- 734
- Forks
- 412
- Merge medio
- 1 d 5 h
- PR fusionados (30 d)
- 8
Descripción
While testing the latest ipywidgets support for passing numpy arrays as binary buffers with metadata, I inadvertedly tried to assign a numpy slice to a `traittypes.Array` trait, and this custom function (lifted from pythreejs) then creates a `memoryview` of a non-contiguous `ndarray`:
```
def array_to_json(value, widget):
return {
'shape': value.shape,
'dtype': str(value.dtype),
'buffer': memoryview(value)
}
```
which shortly after fails inside zmq:
```
ERROR:tornado.general:Uncaught exception, closing connection.
Traceback (most recent call last):
File "/home/martinal/envs/visenv-dev/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "/home/martinal/envs/visenv-dev/lib/python3.5/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/home/martinal/envs/visenv-dev/lib/python3.5/site-packages/ipykernel/iostream.py", line 105, in _handle_event
event_f()
File "/home/martinal/envs/visenv-dev/lib/python3.5/site-packages/ipykernel/iostream.py", line 199, in
self.schedule(lambda : self._really_send(*args, **kwargs))
File "/home/martinal/envs/visenv-dev/lib/python3.5/site-packages/ipykernel/iostream.py", line 207, in _really_send
self.socket.send_multipart(msg, *args, **kwargs)
File "/home/martinal/envs/visenv-dev/lib/python3.5/site-packages/zmq/sugar/socket.py", line 366, in send_multipart
self.send(msg, SNDMORE|flags, copy=copy, track=track)
File "zmq/backend/cython/socket.pyx", line 636, in zmq.backend.cython.socket.Socket.send (zmq/backend/cython/socket.c:7305)
File "zmq/backend/cython/socket.pyx", line 683, in zmq.backend.cython.socket.Socket.send (zmq/backend/cython/socket.c:7048)
File "zmq/backend/cython/socket.pyx", line 188, in zmq.backend.cython.socket._send_copy (zmq/backend/cython/socket.c:2815)
File "zmq/utils/buffers.pxd", line 200, in zmq.utils.buffers.asbuffer_r (zmq/backend/cython/socket.c:8876)
File "zmq/utils/buffers.pxd", line 159, in zmq.utils.buffers.asbuffer (zmq/backend/cython/socket.c:8295)
BufferError: memoryview: underlying buffer is not contiguous
ERROR:tornado.general:Uncaught exception, closing connection.
```
This could be dealt with a number of places:
1) by the user (not very user friendly)
2) by the widget developer (but this is a fairly generic problem)
3) in traittypes.Array.validate, by calling np.ascontiguousarray instead of np.asarray
4) or, as @jasongrout suggests:
> Perhaps the underlying python/zmq interface should use something like PyMemoryView_GetContiguous to get a contiguous buffer.
@minrk any opinions on the matter?
Guía de contribución
Línea de trabajo
Start with traittypes.Array.validate and the array_to_json example, then inspect the zmq buffer path described in the traceback. Compare the proposed user, widget, validation, and PyMemoryView_GetContiguous handling options; done requires an agreed approach and confirmation that non-contiguous arrays no longer fail during buffer transmission.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- jupyter-notebook, numpy, python
- Área
- backend, networking
- Tipo de issue
- Error
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Estado de actividad
- Estancado
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 30/100