ipld / ipld/go-car

Indexes are slurpy and allocates lots of memory

Open
#314 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
175
Forks
54
Avg merge
6d 5h
Merged PRs (30d)
4

Description

Indexes datastructure are fully copied into memory when parsing an index.

There is no limit on how big thoses copies were (because practical indexes used by peoples are bigger than what would be a safe limit).
So if you parse an index from untrusted user input, they can send you a really big index and memory hog or OOM you.

Would be nice if we have a way to parse untrusted indexes without opening yourself to OOMs.

This was first attempted in #312 (see discussion in https://github.com/ipld/go-car-priv/pull/2):
- https://github.com/ipld/go-car/commit/3923d315f1083ce25ee4e790287cc5702a381b13

But was reverted due to API breaking considerations (see discussion in the channel and https://github.com/ipld/go-car-priv/pull/18)):
- https://github.com/ipld/go-car/commit/4bc677484af39d1153e075ae162c5ebcc6e396a3

See https://github.com/ipld/go-car/security/advisories/GHSA-9x4h-8wgm-8xfg
> *go-car@v2.4.0* also includes additional documentation regarding the dangers of consuming CARv2 index data from untrusted sources and a recommendation to regenerate indexes of CAR data from such sources where an index is required.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.