daphne-project / daphne-project/daphne
Extend and optimize data transfer between Python and DAPHNE
- Dominant language
- C++
- Stars
- 81
- Forks
- 83
- PR merge metrics
- No merged PRs in 30d
Description
**Motivation:** DAPHNE offers its own domain-specific language DaphneDSL for linear and relational algebra. However, the most popular language for data science nowadays is Python, and data scientists cannot be expected to switch to DaphneDSL. Therefore, DAPHNE offers a Python API, the so-called DaphneLib. DaphneLib offers DAPHNE’s operations in Python, while still guaranteeing DAPHNE’s unique performance characteristics through lazy evaluation. To allow a seamless integration of DAPHNE into the existing Python-based data science ecosystem, DaphneLib supports efficient bi-directional data exchange with established Python libraries for data science, such as numpy, pandas, TensorFlow, and PyTorch. However, the existing implementation has some limitations, e.g., focus on numeric data types and a small range of data structures. Furthermore, a systematic and repeatable performance evaluation is still missing.
**Task:** This project is about benchmarking, optimizing, and extending the existing data transfer features of DaphneLib. It can be split into the following main sub-tasks:
1. A **systematic evaluation of DaphneLib’s data transfer features** to find out if the data transfer with all hitherto supported Python libraries works correctly (e.g., does the correct data arrive at the receiver, is the data still intact at the sender, etc.) and efficiently (e.g., is it really zero-copy or are there hidden overheads) in both directions (Python to DAPHNE, DAPHNE to Python). The dimensions of this systematic evaluation include:
- different Python data structures: `numpy.array`, `pandas.DataFrame` (including sparse and categorical dtypes), `pandas.Series`, `tensorflow.tensor`, `torch.tensor`, with the corresponding DAPHNE data types (`DenseMatrix` and `Frame`)
- different value types: floating-point numbers (64 and 32 bits), signed/unsigned integers (64, 32, 16, 8 bits)
- different shapes (including 0-lengths; 1d, 2d, >2d objects)
- transferring DAPHNE views into matrices/frames from DAPHNE to Python
In case the evaluation reveals bugs or performance bottlenecks, these should be fixed in the course of the project (depending on their complexity). The systematic tests should become a part of DAPHNE’s test suite (testing for correctness and ideally even for zero-copy transfer).
1. **Extending DaphneLib’s data transfer features** by the following:
- *String data transfer:* So far, DaphneLib can only transfer numeric value types. However, we also need to be able to transfer string data. For instance, it should be possible to transfer numpy arrays of string dtype and pandas DataFrames with string columns to DAPHNE and back. To this end, find out which internal data layout is used by the mentioned Python libraries. If possible, adopt a zero-copy approach. Otherwise, try to make any necessary conversions and transfer as efficient as possible.
- *Support for more Python data structures:* While DaphneLib already supports several important Python data structures, there are still some missing, e.g.:
- Ordinary Python lists: Supporting these would be super useful for writing test cases (currently, there is no simple way to have a small example matrix of arbitrary values in DaphneLib). It should be possible to do something like `dc.from_python([1, 2, 3])`
- `numpy.vector` (see also #616)
- `pandas.Series` (currently done via conversion to `DataFrame`; if this is inefficient, maybe it can be improved)
- pandas categorical dtype (currently converted to standard `DataFrame`; if this is inefficient, maybe it can be improved)
- Sparse matrices: support for efficient data transfer between structures like `scipy.sparse.csr_matrix`, `pandas.arrays.SparseArray`, `tensorflow.sparse.SparseTensor` and DAPHNE’s `CSRMatrix`
- Optionally the data structures of other Python libraries from fields such as image processing (e.g., Pillow/PIL) or audio. These should be optional dependencies like TensorFlow/PyTorch are.
1. **End-to-end experiments showcasing the DaphneLib’s data transfer** in comparison to other systems than can interact with Python libraries. Here, one could implement some simple integrated data analysis pipelines partly in Python and partly in DAPHNE (or baselines). An example could be reading a data set in Python, transferring it to DAPHNE and processing it there using existing algorithm implementations (e.g., decision trees, linear regression), and transferring the results back to Python, maybe even plotting them in Python.
**Hints:**
- Get started with DAPHNE
- Set up your DAPHNE [development environment](https://daphne-eu.github.io/daphne/GettingStarted/#quickstart-for-developers) and [run the test suite](https://daphne-eu.github.io/daphne/development/Testing/).
- Read and follow the [contribution guidelines](https://daphne-eu.github.io/daphne/development/Contributing/).
- Have a look at the DAPHNE documentation, especially the pages on [DaphneLib](https://daphne-eu.github.io/daphne/DaphneLib/Overview/)
- Get familiar with the data models and in-memory data representations of DAPHNE and other data science libraries, such as numpy, pandas, TensorFlow, and PyTorch, and.
- Have a look at the DAPHNE codebase, e.g., the DaphneLib implementation in `src/api/python/`, and the entry point to the C++ part of DAPHNE in `src/api/cli/daphnelib/daphnelib.cpp`; to name just a few starting points.
- Work on some good first issues related to this project, e.g., #710, #715, and/or #718. That way, you can get familiar with development and open-source processes in DAPHNE with some conceptually simple tasks.
- All new features should be easily maintainable, i.e., they should be (1) covered by meaningful test cases, and (2) documented (developer docs page explaining the overall design and source code comments).
- Split your contribution into multiple pull requests.
Contributor guide
Research direction
Start by reading the DaphneLib implementation in src/api/python/ and the C++ entry point at src/api/cli/daphnelib/daphnelib.cpp, then run the test suite after setting up the development environment. Done means a scoped set of data-transfer benchmarks and tests covering the selected structures, types, shapes, and directions, with any resulting fixes documented and covered by meaningful tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, numpy, pandas, python, pytorch, tensorflow
- Domain
- api, data, performance, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100