HumanSignal / HumanSignal/label-studio

Allow for Binary (Feather) Files as Timeseries Tasks.

Open
#2,889 0 comments 1 reaction 1 assignee Claimed by @makseq View on GitHub
feature import time series
Dominant language
TypeScript
Stars
28.3k
Forks
3.7k
Avg merge
14h
Merged PRs (30d)
15

Description

Labelling large timeseries datasets using label studio currently has the limitation of expanding already large binary formats into even larger text files. This limits the amount of data you can host, more than is necessary, as well as increasing the file read and data parsing time (although I don't know how much of the time loading a timeseries task is spent reading and parsing files).

**Describe the solution you'd like**
Allow for timeseries tasks to point at [feather files](https://arrow.apache.org/docs/python/feather.html) and have them loaded as [Dataframe](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.htm) objects through pandas' API, in the same way [CSVs are currently loaded](https://github.com/heartexlabs/label-studio/blob/6c17fa1bba2b500f3a14c943fae2b331a3237c4b/label_studio/data_import/models.py#L75) through [`pandas.read_csv`](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html), feather files can be loaded with [`pandas.read_feather`](https://pandas.pydata.org/docs/reference/api/pandas.read_feather.html).

**Describe alternatives you've considered**
Settling on feather was largely informed by this ['towardsdatascience' article](https://towardsdatascience.com/the-best-format-to-save-pandas-data-414dca023e0d). HDF5 was also considered due to its wide acceptance and multi-language support, and we may want to allow HDF5 files as well, but it does [have its critics](https://cyrille.rossant.net/moving-away-hdf5/).

Beyond criticisms that likely also apply to feather, HDF5 is built to handle and chunk data too large to fit in RAM, which is overkill for label-studio which at this point operates best with smaller size chunks (based on my user experience). And for all the extra features we don't need from HDF5 comes an overhead best avoided as detailed in the ['towardsdatascience' article](https://towardsdatascience.com/the-best-format-to-save-pandas-data-414dca023e0d) already mentioned.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.