aimclub / aimclub/FEDOT

refactor: introduce torch-friendly data model

Open
#1,416 7 comments 0 reactions 0 assignees View on GitHub
api enhancement help wanted refactoring
Dominant language
Python
Stars
712
Forks
96
Avg merge
3d 1h
Merged PRs (30d)
10

Description

## Summary

Introduce `TensorData` and `TensorOutputData` classes to unify data representation around `torch.Tensor`, and update preprocessing and model interfaces to support batched, multi-dimensional data.

## Motivation

Current `InputData/OutputData` are tied to `NumPy/Pandas`, which limits performance and scalability for deep learning models. A tensor-first approach will better support GPU acceleration, batch processing, and complex data types.

## High-level architecture
* Core tensor types:
* `TensorData` (aka `TensorBatch`): primary input container
* `TensorOutputData` (aka `TensorBatchOut`): primary prediction/output container
* Backend & dataloader integration:
* `TensorData.backend_name` + `backend property` (torch-only initially)
* `TensorData.dataset` / `TensorData.dataloader` via `TensorDataset` / `DataLoader`, configurable by `dataloader_kwargs`
* Legacy shim layer:
* Adapters: `InputData` <-> `TensorData`, `OutputData` <-> `TensorOutputData`
* Old APIs (`fit(InputData)`, `predict(InputData)` -> `OutputData`) call into new tensor-based implementations
* Transforms & multi-source layer:
* Preprocessing/feature-engineering as separate transforms acting on `TensorData`
* Slim `TensorMultiData` (mapping `str` -> `TensorData`) replaces most of `MultiModalData` + `DataMerger`

## Reference-level explanation

* Merge [fedot_ind data converters](https://github.com/aimclub/Fedot.Industrial/blob/80e9d295c27b774d53e5c734b252168998537680/fedot_ind/core/architecture/preprocessing/data_convertor.py) and [fedot_ind decorators](https://github.com/aimclub/Fedot.Industrial/blob/main/fedot_ind/core/architecture/abstraction/decorators.py) into `fedot/core/data/*`
* Ensure shape preservation and dtype/device consistency
* (Optionally) Add batch iteration API: `iter_batches(batch_size, ...)`
* Write unit tests for conversion and training loops

## Drawbacks

Adds complexity to data handling layers (but will be refactored anyway)

## Unresolved Questions

Should we consider adding more backends in the future? This might enhance scalability.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.