indygreg / indygreg/python-zstandard

Segfault in `ZstdCompressorIterator_iternext` with buffer-protocol data

Open
#303 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
642
Forks
116
Avg merge
1d 14h
Merged PRs (30d)
5

Description

Hi,

I've been experimenting with passing `memoryview`s into `read_to_iter`, and I've been hitting segfaults pretty regularly on large pieces of data.

The offending line is `Py_DECREF(self->readResult)` in `compressoriterator.c`. For input data passed in as buffer objects, this pointer (`self->readResult`) is never not-null.

I've been unable to create a small re-producer which hits these conditions:

```c
if (self->input.pos < self->input.size) {
...

if (self->input.pos == self->input.size) {
...
Py_DECREF(self->readResult);
...
}
```

But I'll keep trying.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in compressoriterator.c at ZstdCompressorIterator_iternext and inspect the Py_DECREF(self->readResult) path when read_to_iter receives buffer-protocol data such as memoryview. Work toward a small reproducer for large inputs, then verify that the reproducer no longer segfaults and that the completed iteration remains safe.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.