No streaming encode/decode — entire data loaded into memory
- Dominant language
- Python
- Stars
- 13
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
All codecs load entire data into memory during encode/decode. Go's codecs use `io.Writer`/`io.Reader` for streaming, which is important for large blocks.
### Problem
Current codec interface:
```python
class BlockCodec(BlockEncoder, BlockDecoder):
@abc.abstractmethod
def encode(self, node: IPLDNode) -> bytes:
"""Encode an IPLD data model value into bytes."""
@abc.abstractmethod
def decode(self, data: bytes) -> IPLDNode:
"""Decode bytes into an IPLD data model value."""
```
This requires the entire encoded/decoded data to fit in memory. For large blocks (e.g., multi-megabyte files), this is inefficient.
Go's interface:
```go
type Encoder func(datamodel.Node, io.Writer) error
type Decoder func(datamodel.NodeAssembler, io.Reader) error
```
### Proposed Solution
Add optional streaming methods to `BlockCodec`:
```python
class BlockCodec(BlockEncoder, BlockDecoder):
# ... existing methods ...
def encode_to_stream(self, node: IPLDNode, stream: BinaryIO) -> int:
"""Encode to a stream. Returns bytes written.
Default implementation encodes to bytes then writes.
Subclasses may override for true streaming.
"""
data = self.encode(node)
return stream.write(data)
def decode_from_stream(self, stream: BinaryIO) -> IPLDNode:
"""Decode from a stream.
Default implementation reads all bytes then decodes.
Subclasses may override for true streaming.
"""
data = stream.read()
return self.decode(data)
```
This provides a streaming interface while maintaining backward compatibility. Codecs that can stream efficiently (e.g., DAG-CBOR) can override these methods.
### Related
- Go implementation: go-ipld-prime `codec/api.go`
- File: `dag/codec.py`
Contributor guide
Research direction
Start in dag/codec.py by reading the BlockCodec, BlockEncoder, and BlockDecoder interfaces and the existing codec implementations. Compare the proposed stream methods with the current encode/decode contract and check how codecs are used. Done means callers have a backward-compatible streaming interface, with default behavior preserved for codecs that do not implement true streaming.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 67/100