Support for lists for supported encodings
Open
@knighton is already working on this.
Since Mar 24, 2023.
enhancement
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
🚀 Feature Request
The current supported encodings are listed here: https://github.com/mosaicml/streaming/blob/59f6ec5f8f97cc5f9a75954fef4bef3221460ff8/streaming/base/format/mds/encodings.py#L270
I would like to have support for lists, i.e. columns that are lists of integers, jpegs, etc. Theses list can be of arbitrary lengths.
Motivation
This is particularly relevant for interleaved datasets (think documents that are a sequence of either text or images).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.