Is it possible to estimate pipelines resource usage by input shape so we can scheduler tasks smarter?

Hello,

I am using alot of tpot

Since my data is huge(about 200k*100),I try to use tpot with dask on kubernetes,I find its alway OOM in the middle of training.Every pod I use 32G memory and 8 cores

My guess is that for the same shape input, some pipelines use less memory,so dask works fine with it,while some other pipelines use much more memory,the case OOM.

So I am thinking is it possible to estimate pipelines resource usage,so if a pipeline will use more resource than our configs,then the pipeline just died and score zero.

Since the normal agriothm all have Space Complexity，like O(mn^2)，in my naive guess,this is possible.

Would you please give some advise?Now I am trying is using pipelines with light memory usage