daphne-project / daphne-project/daphne
Efficient File I/O Plug-ins for Widely-used File Formats
- Dominant language
- C++
- Stars
- 81
- Forks
- 83
- PR merge metrics
- No merged PRs in 30d
Description
**Motivation:** The input to integrated data analysis pipelines, that combine query processing, machine learning, and high-performance computing, could be provided in various general-purpose and domain-specific file formats. To embrace a large variety of such formats, DAPHNE is extensible w.r.t. to file readers/writers (pull request coming soon), i.e., expert users can add their own file I/O plug-ins without touching the source code of DAPHNE. However, so far there is only a limited number of plug-ins available.
**Task:** (in C++) Implement additional file I/O plug-ins for a range of widely used file formats for different data modalities, such as tabular data, (sparse) matrices/graphs, audio, images, and time series. These plug-ins may be based on existing open-source libraries (with compatible license). Apply format-specific and format-agnostic tricks to make the file I/O efficient, e.g., by exploiting parallelism or pushing down certain operations into the readers.
**Hints:**
*coming soon*
Contributor guide
Research direction
Start by reviewing DAPHNE's file reader/writer plug-in extension described in the issue and identify the supported plug-in entry points. Choose a specific widely used format and data modality, then verify that the added plug-in reads or writes it efficiently using compatible open-source libraries; completion should include the plug-in and evidence of efficient file I/O.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100