CarperAI / CarperAI/Code-Pile

Intermediate Data Storage Format

Open
#13 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
110
Forks
32
PR merge metrics
No merged PRs in 30d

Description

Questions:
* How do we want to store data in an intermediate format before moving it to the lm_dataformat that uses json lists?
* Do we even want an intermediate data format?

Let's use this issue to discuss this topic.

Resources:
* https://www.databricks.com/glossary/what-is-parquet#:~:text=What%20is%20Parquet%3F,handle%20complex%20data%20in%20bulk.
* https://arrow.apache.org/

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.