Allow user to change global metrics of autoscaling in ConfigMap: config-autoscaler

Describe the feature

Allow users to change global metrics of autoscaling in ConfigMap: config-autoscaler.

It seems that the global metric of autoscaling is concurrency, because global configurations about concurrency in ConfigMap: config-autoscaler work, e.g., container-concurrency-target-default and container-concurrency-target-percentage.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-autoscaler
  namespace: knative-serving
  labels:
    serving.knative.dev/release: "v0.22.1"
data:
  allow-zero-initial-scale: "false"
  container-concurrency-target-default: "100"
  container-concurrency-target-percentage: "0.7"

And the configuration about rps, requests-per-second-target-default, doesn't work unless autoscaling.knative.dev/metric: "rps" is configured in the InferenceService.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-autoscaler
  namespace: knative-serving
  labels:
    serving.knative.dev/release: "v0.22.1"
data:
  allow-zero-initial-scale: "false"
  requests-per-second-target-default: "100"

apiVersion: serving.kubeflow.org/v1beta1
kind: InferenceService
metadata:
  annotations:
    "sidecar.istio.io/inject": "false"
    # RPS
    autoscaling.knative.dev/metric: "rps"
    # autoscaling.knative.dev/target: "2"
  name: autoscaler-test
  namespace: test
spec:
  predictor:
    canaryTrafficPercent: 100
    serviceAccountName: sa
    tensorflow:
      image: tensorflow/serving:2.4.0
      name: kfserving-container
      runtimeVersion: 2.4.0
      storageUri: s3://tfx/models

If we can set metrics in the ConfigMap for autoscaling. E.g., rps, we don't need to config it every time when creating InferenceServices. It could be a useful feature.

Start with the config-autoscaler ConfigMap settings for container-concurrency-target-default and requests-per-second-target-default, then trace how the autoscaling metric and its defaults are selected. Review the autoscaling.knative.dev/metric annotation examples and relevant autoscaler tests. Done means a metric such as rps can be set globally in the ConfigMap and individual InferenceServices no longer need to repeat it.