refactor: introduce torch-friendly data model
- Dominant language
- Python
- Stars
- 712
- Forks
- 96
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 10
Description
## Summary
Introduce `TensorData` and `TensorOutputData` classes to unify data representation around `torch.Tensor`, and update preprocessing and model interfaces to support batched, multi-dimensional data.
## Motivation
Current `InputData/OutputData` are tied to `NumPy/Pandas`, which limits performance and scalability for deep learning models. A tensor-first approach will better support GPU acceleration, batch processing, and complex data types.
## High-level architecture
* Core tensor types:
* `TensorData` (aka `TensorBatch`): primary input container
* `TensorOutputData` (aka `TensorBatchOut`): primary prediction/output container
* Backend & dataloader integration:
* `TensorData.backend_name` + `backend property` (torch-only initially)
* `TensorData.dataset` / `TensorData.dataloader` via `TensorDataset` / `DataLoader`, configurable by `dataloader_kwargs`
* Legacy shim layer:
* Adapters: `InputData` <-> `TensorData`, `OutputData` <-> `TensorOutputData`
* Old APIs (`fit(InputData)`, `predict(InputData)` -> `OutputData`) call into new tensor-based implementations
* Transforms & multi-source layer:
* Preprocessing/feature-engineering as separate transforms acting on `TensorData`
* Slim `TensorMultiData` (mapping `str` -> `TensorData`) replaces most of `MultiModalData` + `DataMerger`
## Reference-level explanation
* Merge [fedot_ind data converters](https://github.com/aimclub/Fedot.Industrial/blob/80e9d295c27b774d53e5c734b252168998537680/fedot_ind/core/architecture/preprocessing/data_convertor.py) and [fedot_ind decorators](https://github.com/aimclub/Fedot.Industrial/blob/main/fedot_ind/core/architecture/abstraction/decorators.py) into `fedot/core/data/*`
* Ensure shape preservation and dtype/device consistency
* (Optionally) Add batch iteration API: `iter_batches(batch_size, ...)`
* Write unit tests for conversion and training loops
## Drawbacks
Adds complexity to data handling layers (but will be refactored anyway)
## Unresolved Questions
Should we consider adding more backends in the future? This might enhance scalability.
Contributor guide
Assessment
This issue has not been assessed yet.