aiondemand / aiondemand/aiondemand

[ENH] API validator classes

Đang mở
#63 5 bình luận 1 reaction 1 người được giao Được @geetu040 nhận Xem trên GitHub
enhancement
Ngôn ngữ chính
Python
Star
38
Fork
89
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

I think it would be good to have API contracts as first-class citizens in order to have an explicit, extensible reference point for unified APIs.

Idea: having a concept of different API contracts explicitply in aiod (first class citizen)

Examples:

* sklearn classifier
* sktime forecaster
* sktime TSclassifier
* pytorch-tabular tabular torch model (not sklearn but one layer below)

Requirements:

* API contract is a first class citizen following strategy pattern
* taggable and retrievable
* contract C can be paired with AI algorithm asset A to check "A satisfies C"

Tentative usage mock design:

```python
from sklearn.ensemble import RandomForestClassifier

from aiod.contracts import SklearnClassifier

SklearnClassifier.istypeof(RandomForestClassifier)
# True
# "RandomForestClassifier is an sklearn classifier"

SklearnRegressor.istypeof(RandomForestClassifier)
# False

SklearnClassifier.runtests(RandomForestClassifier)
# runs tests
# returns test summary or printout

# there are two ways to invoke:
# prior knowledge, i.e., tested somewhere and available as database information (e.g., lookup through batteries included clasess)
# actually re-running the tests, via runtests
```

Methods that all "API contracts" have:

* `istypeof(cls)` - returns `True` or `False`
* `runtests` - runs tests for contract

We can change names of course.

Use cases:

* use case 1 - for automation, e.g.,
* run CRON checks on whether estimator A still satisfies a given contract
* run type and API checks on custom estimators
* this ensures API conformance of estimators offered by the platform, i.e., users can trust if the API contract is "green" they can plug into their workflows
* use case 2: plugin developers for their own estimators
* single endpoint for API contract checking for all AI
* (as long as the API contract is being maintained)
* "API checking service"

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.