automl / automl/Auto-PyTorch

Score function error on example code

Open
#484 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
300
PR merge metrics
No merged PRs in 30d

Description

NOTE: ISSUES ARE NOT FOR CODE HELP - Ask for Help at https://stackoverflow.com

Your issue may already be reported!
Also, please search on the [issue tracker](../) before creating one.

* **I'm submitting a ...**
- [x] bug report
- [ ] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.

# Issue Description
When running the example code provided on the README file, the `api.score` function throws an exception during an internal call due to missing arguments.

## Expected Behavior
A score value to be calculated based on the predictions and actual values.

## Current Behavior
A TypeError is thrown due to an internal call in the `score` function.

## Your Code

```python
from autoPyTorch.api.time_series_forecasting import TimeSeriesForecastingTask

# data and metric imports
from sktime.datasets import load_longley
targets, features = load_longley()

# define the forecasting horizon
forecasting_horizon = 3

# Dataset optimized by APT-TS can be a list of np.ndarray/ pd.DataFrame where each series represents an element in the
# list, or a single pd.DataFrame that records the series
# index information: to which series the timestep belongs? This id can be stored as the DataFrame's index or a separate
# column
# Within each series, we take the last forecasting_horizon as test targets. The items before that as training targets
# Normally the value to be forecasted should follow the training sets
y_train = [targets[: -forecasting_horizon]]
y_test = [targets[-forecasting_horizon:]]

# same for features. For uni-variant models, X_train, X_test can be omitted and set as None
X_train = [features[: -forecasting_horizon]]
# Here x_test indicates the 'known future features': they are the features known previously, features that are unknown
# could be replaced with NAN or zeros (which will not be used by our networks). If no feature is known beforehand,
# we could also omit X_test
known_future_features = list(features.columns)
X_test = [features[-forecasting_horizon:]]

start_times = [targets.index.to_timestamp()[0]]
freq = '1Y'

# initialise Auto-PyTorch api
api = TimeSeriesForecastingTask()

# Search for an ensemble of machine learning algorithms
api.search(
X_train=X_train,
y_train=y_train,
X_test=X_test,
optimize_metric='mean_MAPE_forecasting',
n_prediction_steps=forecasting_horizon,
memory_limit=16 * 1024, # Currently, forecasting models use much more memories
freq=freq,
start_times=start_times,
func_eval_time_limit_secs=50,
total_walltime_limit=60,
min_num_test_instances=1000, # proxy validation sets. This only works for the tasks with more than 1000 series
known_future_features=known_future_features,
)

# our dataset could directly generate sequences for new datasets
test_sets = api.dataset.generate_test_seqs()

# Calculate test accuracy
y_pred = api.predict(test_sets)
score = api.score(y_pred, y_test)
print("Forecasting score", score)
```

## Error message

```
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
in
52 # Calculate test accuracy
53 y_pred = api.predict(test_sets)
---> 54 score = api.score(y_pred, y_test)
55 print("Forecasting score", score)

/usr/local/lib/python3.8/dist-packages/autoPyTorch/api/base_task.py in score(self, y_pred, y_test)
1908 raise ValueError("AutoPytorch failed to infer a task type from the dataset "
1909 "Please check the log file for related errors. ")
-> 1910 return calculate_score(target=y_test, prediction=y_pred,
1911 task_type=STRING_TO_TASK_TYPES[self.task_type],
1912 metrics=[self._metric])

/usr/local/lib/python3.8/dist-packages/autoPyTorch/pipeline/components/training/metrics/utils.py in calculate_score(target, prediction, task_type, metrics, **score_kwargs)
144 score_dict[metric_.name] = metric_._sign * metric_(target_scaled, cprediction_scaled, **score_kwargs)
145 else:
--> 146 score_dict[metric_.name] = metric_._sign * metric_(target, cprediction, **score_kwargs)
147 elif task_type in REGRESSION_TASKS:
148 cprediction = sanitize_array(prediction)

TypeError: __call__() missing 2 required positional arguments: 'sp' and 'n_prediction_steps'
```

## Local environment
* Google Colab Compute Engine with GPU
* Python 3.8.15

pip freeze

```
absl-py==1.3.0
aeppl==0.0.33
aesara==2.7.9
aiohttp==3.8.3
aiosignal==1.3.1
alabaster==0.7.12
albumentations==1.2.1
alembic==1.8.1
altair==4.2.0
appdirs==1.4.4
arviz==0.12.1
astor==0.8.1
astropy==4.3.1
astunparse==1.6.3
async-timeout==4.0.2
asynctest==0.13.0
atari-py==0.2.9
atomicwrites==1.4.1
attrs==22.1.0
audioread==3.0.0
autograd==1.5
autopage==0.5.1
autoPyTorch==0.2.1
Babel==2.11.0
backcall==0.2.0
beautifulsoup4==4.6.3
bleach==5.0.1
blis==0.7.9
bokeh==2.3.3
branca==0.6.0
bs4==0.0.1
CacheControl==0.12.11
cached-property==1.5.2
cachetools==5.2.0
catalogue==2.0.8
catboost==1.1.1
certifi==2022.9.24
cffi==1.15.1
cftime==1.6.2
chardet==3.0.4
charset-normalizer==2.1.1
click==8.1.3
cliff==4.1.0
clikit==0.6.2
cloudpickle==2.2.0
cmaes==0.9.0
cmake==3.22.6
cmd2==2.4.2
cmdstanpy==1.0.8
colorcet==3.0.1
colorlog==6.7.0
colorlover==0.3.0
community==1.0.0b1
confection==0.0.3
ConfigSpace==0.6.0
cons==0.4.5
contextlib2==0.5.5
contourpy==1.0.6
convertdate==2.4.0
crashtest==0.3.1
crcmod==1.7
cufflinks==0.17.3
cupy-cuda11x==11.0.0
cvxopt==1.3.0
cvxpy==1.2.2
cycler==0.11.0
cymem==2.0.7
Cython==0.29.32
daft==0.0.4
dask==2022.12.0
datascience==0.17.5
db-dtypes==1.0.4
debugpy==1.0.0
decorator==4.4.2
defusedxml==0.7.1
Deprecated==1.2.13
descartes==1.1.0
dill==0.3.6
distributed==2022.12.0
dlib==19.24.0
dm-tree==0.1.7
dnspython==2.2.1
docutils==0.17.1
dopamine-rl==1.0.5
earthengine-api==0.1.332
easydict==1.10
ecos==2.0.10
editdistance==0.5.3
emcee==3.1.3
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.1/en_core_web_sm-3.4.1-py3-none-any.whl
entrypoints==0.4
ephem==4.1.3
et-xmlfile==1.1.0
etils==0.9.0
etuples==0.3.8
fa2==0.3.5
fastai==2.7.10
fastcore==1.5.27
fastdownload==0.0.7
fastdtw==0.3.4
fastjsonschema==2.16.2
fastprogress==1.0.3
fastrlock==0.8.1
feather-format==0.4.1
filelock==3.8.0
fire==0.4.0
firebase-admin==5.3.0
fix-yahoo-finance==0.0.22
flaky==3.7.0
Flask==1.1.4
flatbuffers==1.12
folium==0.12.1.post1
fonttools==4.38.0
frozenlist==1.3.3
fsspec==2022.11.0
future==0.16.0
gast==0.4.0
GDAL==2.2.2
gdown==4.4.0
gensim==3.6.0
geographiclib==1.52
geopy==1.17.0
gin-config==0.5.0
glob2==0.7
gluonts==0.11.3
google==2.0.3
google-api-core==2.8.2
google-api-python-client==1.12.11
google-auth==2.15.0
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.6
google-cloud-bigquery==3.3.6
google-cloud-bigquery-storage==2.16.2
google-cloud-core==2.3.2
google-cloud-datastore==2.9.0
google-cloud-firestore==2.7.2
google-cloud-language==2.6.1
google-cloud-storage==2.5.0
google-cloud-translate==3.8.4
google-colab @ file:///colabtools/dist/google-colab-1.0.0.tar.gz
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.4.0
googleapis-common-protos==1.57.0
googledrivedownloader==0.4
graphviz==0.20.1
greenlet==2.0.1
grpcio==1.51.1
grpcio-status==1.48.2
gspread==3.4.2
gspread-dataframe==3.0.8
gym==0.25.2
gym-notices==0.0.8
h5py==3.1.0
HeapDict==1.0.1
hijri-converter==2.2.4
holidays==0.17
holoviews==1.14.9
html5lib==1.0.1
httpimport==0.5.18
httplib2==0.17.4
httpstan==4.6.1
humanize==0.5.1
hyperopt==0.1.2
idna==3.4
imageio==2.22.4
imagesize==1.4.1
imbalanced-learn==0.8.1
imblearn==0.0
imgaug==0.4.0
importlib-metadata==5.1.0
importlib-resources==5.10.0
imutils==0.5.4
inflect==2.1.0
intel-openmp==2022.2.1
intervaltree==2.1.0
ipykernel==5.3.4
ipython==7.9.0
ipython-genutils==0.2.0
ipython-sql==0.3.9
ipywidgets==7.7.1
itsdangerous==1.1.0
jax==0.3.25
jaxlib @ https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.3.25+cuda11.cudnn805-cp38-cp38-manylinux2014_x86_64.whl
jieba==0.42.1
Jinja2==3.1.2
joblib==1.2.0
jpeg4py==0.1.4
jsonschema==4.3.3
jupyter-client==6.1.12
jupyter-console==6.1.0
jupyter-core==4.11.2
jupyterlab-widgets==3.0.3
kaggle==1.5.12
kapre==0.3.7
keras==2.9.0
Keras-Preprocessing==1.1.2
keras-vis==0.4.1
kiwisolver==1.4.4
korean-lunar-calendar==0.3.1
langcodes==3.3.0
libclang==14.0.6
librosa==0.8.1
lightgbm==3.3.3
lightning-utilities==0.3.0
llvmlite==0.39.1
lmdb==0.99
locket==1.0.0
lockfile==0.12.2
logical-unification==0.4.5
LunarCalendar==0.0.9
lxml==4.9.1
Mako==1.2.4
Markdown==3.4.1
MarkupSafe==2.1.1
marshmallow==3.19.0
matplotlib==3.6.2
matplotlib-venn==0.11.7
miniKanren==1.0.3
missingno==0.5.1
mistune==0.8.4
mizani==0.7.3
mkl==2019.0
mlxtend==0.14.0
more-itertools==9.0.0
moviepy==0.2.3.5
mpmath==1.2.1
msgpack==1.0.4
multidict==6.0.3
multipledispatch==0.6.0
multitasking==0.0.11
murmurhash==1.0.9
music21==5.5.0
natsort==5.5.0
nbconvert==5.6.1
nbformat==5.7.0
netCDF4==1.6.2
networkx==2.8.8
nibabel==3.0.2
nltk==3.7
notebook==5.7.16
numba==0.56.4
numexpr==2.8.4
numpy==1.22.4
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
oauth2client==4.1.3
oauthlib==3.2.2
okgrade==0.4.3
opencv-contrib-python==4.6.0.66
opencv-python==4.6.0.66
opencv-python-headless==4.6.0.66
openpyxl==3.0.10
opt-einsum==3.3.0
optuna==2.10.1
osqp==0.6.2.post0
packaging==21.3
palettable==3.3.0
pandas==1.5.2
pandas-datareader==0.9.0
pandas-gbq==0.17.9
pandas-profiling==1.4.1
pandocfilters==1.5.0
panel==0.12.1
param==1.12.2
parso==0.8.3
partd==1.3.0
pastel==0.2.1
pathlib==1.0.1
pathy==0.9.0
patsy==0.5.3
pbr==5.11.0
pep517==0.13.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.3.0
pip-tools==6.2.0
plotly==5.11.0
plotnine==0.8.0
pluggy==0.7.1
pooch==1.6.0
portpicker==1.3.9
prefetch-generator==1.0.3
preshed==3.0.8
prettytable==3.5.0
progressbar2==3.38.0
prometheus-client==0.15.0
promise==2.3
prompt-toolkit==2.0.10
prophet==1.1.1
proto-plus==1.22.1
protobuf==3.20.1
psutil==5.9.4
psycopg2==2.9.5
ptyprocess==0.7.0
py==1.11.0
pyarrow==9.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.6
pycparser==2.21
pyct==0.4.8
pydantic==1.10.2
pydata-google-auth==1.4.0
pydot==1.3.0
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
pyemd==0.5.1
pyerfa==2.0.0.1
Pygments==2.6.1
pygobject==3.26.1
pylev==1.4.0
pymc==4.1.4
PyMeeus==0.5.11
pymongo==4.3.3
pymystem3==0.2.0
pynisher==0.6.4
PyOpenGL==3.1.6
pyparsing==3.0.9
pyperclip==1.8.2
pyrfr==0.8.3
pyrsistent==0.19.2
pysimdjson==3.2.0
pysndfile==1.3.8
PySocks==1.7.1
pystan==3.3.0
pytest==3.6.4
python-apt==0.0.0
python-dateutil==2.8.2
python-louvain==0.16
python-slugify==7.0.0
python-utils==3.4.5
pytorch-forecasting==0.10.3
pytorch-lightning==1.8.3.post1
pytz==2022.6
pyviz-comms==2.2.1
PyWavelets==1.4.1
PyYAML==6.0
pyzmq==23.2.1
qdldl==0.1.5.post2
qudida==0.0.4
regex==2022.10.31
requests==2.28.1
requests-oauthlib==1.3.1
resampy==0.4.2
rpy2==3.5.5
rsa==4.9
scikit-image==0.19.3
scikit-learn==0.24.2
scipy==1.9.3
screen-resolution-extra==0.0.0
scs==3.2.2
seaborn==0.11.2
Send2Trash==1.8.0
setuptools-git==1.2
Shapely==1.8.5.post1
six==1.16.0
sklearn-pandas==1.8.0
sktime==0.14.1
smac==1.4.0
smart-open==5.2.1
snowballstemmer==2.2.0
sortedcontainers==2.4.0
soundfile==0.11.0
spacy==3.4.3
spacy-legacy==3.0.10
spacy-loggers==1.0.3
Sphinx==1.8.6
sphinxcontrib-serializinghtml==1.1.5
sphinxcontrib-websupport==1.2.4
SQLAlchemy==1.4.44
sqlparse==0.4.3
srsly==2.4.5
statsmodels==0.13.5
stevedore==4.1.1
sympy==1.7.1
tables==3.7.0
tabulate==0.9.0
tblib==1.7.0
tenacity==8.1.0
tensorboard==2.11.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorboardX==2.5.1
tensorflow==2.9.2
tensorflow-datasets==4.6.0
tensorflow-estimator==2.9.0
tensorflow-gcs-config==2.9.1
tensorflow-hub==0.12.0
tensorflow-io-gcs-filesystem==0.28.0
tensorflow-metadata==1.11.0
tensorflow-probability==0.17.0
termcolor==2.1.1
terminado==0.13.3
testpath==0.6.0
text-unidecode==1.3
textblob==0.15.3
thinc==8.1.5
threadpoolctl==3.1.0
tifffile==2022.10.10
toml==0.10.2
tomli==2.0.1
toolz==0.12.0
torch==1.13.0
torchaudio @ https://download.pytorch.org/whl/cu113/torchaudio-0.12.1%2Bcu113-cp38-cp38-linux_x86_64.whl
torchmetrics==0.11.0
torchsummary==1.5.1
torchtext==0.13.1
torchvision==0.14.0
tornado==6.2
tqdm==4.64.1
traitlets==5.1.1
tweepy==3.10.0
typeguard==2.7.1
typer==0.7.0
typing-extensions==4.4.0
tzlocal==1.5.1
uritemplate==3.0.1
urllib3==1.26.13
vega-datasets==0.9.0
wasabi==0.10.1
wcwidth==0.2.5
webargs==8.2.0
webencodings==0.5.1
Werkzeug==2.2.2
widgetsnbextension==3.6.1
wordcloud==1.8.2.2
wrapt==1.14.1
xarray==0.20.2
xarray-einstats==0.2.2
xgboost==0.90
xkit==0.0.0
xlrd==1.1.0
xlwt==1.3.0
yarl==1.8.2
yellowbrick==1.5
zict==2.2.0
zipp==3.11.0
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.