ipld / ipld/py-ipld-dag

No streaming encode/decode — entire data loaded into memory

Open
#32 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
13
Forks
9
PR merge metrics
No merged PRs in 30d

Description

All codecs load entire data into memory during encode/decode. Go's codecs use `io.Writer`/`io.Reader` for streaming, which is important for large blocks.

### Problem

Current codec interface:

```python
class BlockCodec(BlockEncoder, BlockDecoder):
@abc.abstractmethod
def encode(self, node: IPLDNode) -> bytes:
"""Encode an IPLD data model value into bytes."""

@abc.abstractmethod
def decode(self, data: bytes) -> IPLDNode:
"""Decode bytes into an IPLD data model value."""
```

This requires the entire encoded/decoded data to fit in memory. For large blocks (e.g., multi-megabyte files), this is inefficient.

Go's interface:

```go
type Encoder func(datamodel.Node, io.Writer) error
type Decoder func(datamodel.NodeAssembler, io.Reader) error
```

### Proposed Solution

Add optional streaming methods to `BlockCodec`:

```python
class BlockCodec(BlockEncoder, BlockDecoder):
# ... existing methods ...

def encode_to_stream(self, node: IPLDNode, stream: BinaryIO) -> int:
"""Encode to a stream. Returns bytes written.

Default implementation encodes to bytes then writes.
Subclasses may override for true streaming.
"""
data = self.encode(node)
return stream.write(data)

def decode_from_stream(self, stream: BinaryIO) -> IPLDNode:
"""Decode from a stream.

Default implementation reads all bytes then decodes.
Subclasses may override for true streaming.
"""
data = stream.read()
return self.decode(data)
```

This provides a streaming interface while maintaining backward compatibility. Codecs that can stream efficiently (e.g., DAG-CBOR) can override these methods.

### Related

- Go implementation: go-ipld-prime `codec/api.go`
- File: `dag/codec.py`

Contributor guide

Open the contributing guide

Research direction

Start in dag/codec.py by reading the BlockCodec, BlockEncoder, and BlockDecoder interfaces and the existing codec implementations. Compare the proposed stream methods with the current encode/decode contract and check how codecs are used. Done means callers have a backward-compatible streaming interface, with default behavior preserved for codecs that do not implement true streaming.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
67/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.