kubectl get <category> (aka category-expansion feature) is lacking documentation, making it hard to discover and use

### What happened?

It is hard for users to
* discover that kubectl support category-expanded resources syntax `kubectl get `
including the special `all` category.
* to discover available categories for use in `kubectl get `
* to understand the expected semantics of `kubectl get all` command

It is hard for CRD authors to find guidance on usage of categories, including the special `all` category

### What did you expect to happen?

Expecting the kubectl usage output and kubectl reference documentation to document:
* the category-expansion feature, along with an introduction to the concept of category, link to the CRD category sections
* which of the core resource types are returned or omitted from using the specific `all` category
* whether the core resource types are assigned
* the limits of the current implementation (server-side and client side) of this feature explaining frequently observed slow performance in presence of many resource types, in particular CRDs. AFAIK, current `kubectl` is: lookup of all CRDs (possibly from kubectl local cache), inspection of each CRD to match against the request categories, then one api call per resource type.
* hint for discovering available category strings that may return resources in a cluster: for ccustom resources resources

In order to bring a consistent user experience when using `kubectl get ` to list custom resources, guidance and conventions could be documented.

### How can we reproduce it (as minimally and precisely as possible)?

---
#### Discovering category-expansion feature in kubectl

Currently https://kubernetes.io/docs/reference/kubectl/generated/kubectl_get/ omits the category argument

> kubectl get [(-o|--output=)json|yaml|kyaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-as-json|jsonpath-file|custom-columns|custom-columns-file|wide] (TYPE[.VERSION][.GROUP] [NAME | -l label] | TYPE[.VERSION][.GROUP]/NAME ...) [flags]

Web search [`"kubectl get all" site:https://kubernetes.io/docs/`](https://www.google.com/search?client=firefox-b-e&channel=entpr&q=%22kubectl+get+all%22+site%3Ahttps%3A%2F%2Fkubernetes.io%2Fdocs%2F) currently returns two matches
* https://kubernetes.io/docs/reference/kubernetes-api/apiextensions/custom-resource-definition-v1/#CustomResourceDefinitionNames
>

categoriesstring array | categories is a list of grouped resources this custom resource belongs to (e.g. 'all'). This is published in API discovery documents, and used by clients to support invocations like `kubectl get all`.
-- | --

https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#categories
> Categories is a list of grouped resources the custom resource belongs to (eg. all). You can use kubectl get to list the resources belonging to the category.
>
> The following example adds all in the list of categories in the CustomResourceDefinition and illustrates how to output the custom resource using kubectl get all.

Searching for ["categories" site:https://kubernetes.io/docs/](https://www.google.com/search?client=firefox-b-e&channel=entpr&q=%22categories%22+site%3Ahttps%3A%2F%2Fkubernetes.io%2Fdocs%2F) additionally returns:

> https://kubernetes.io/docs/reference/kubernetes-api/definitions/api-resource-v1-meta/#APIResource
> APIResource specifies the name of a resource and whether it is namespaced.
>
> Field | Description
> -- | --
> categories string array | categories is a list of the grouped resources this resource belongs to (e.g. 'all')

https://kubernetes.io/docs/reference/kubectl/generated/kubectl_api-resources/ and https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#api-resources

--categories strings
--
  | Limit to resources that belong to the specified categories.

---
#### Discovering categories supported by core resources

No user-facing documentation can be easily found. However, the following documentation for K8S contributors provides useful hints

https://github.com/kubernetes/community/blob/14e2a8d7097fe4fd09b2f4d3e731922fb88c2450/contributors/devel/sig-architecture/api-conventions.md?plain=1#L771-L802

>
> Note: Due to inconsistent behavior when categories collide with resource types,
> and difficulties knowing when it is safe to add new resources to an existing category,
> do not add new categories to built-in resources unless specifically allowed by API reviewers.
> See issues [#7547](https://github.com/kubernetes/kubernetes/issues/7547#issuecomment-355835279)
> [#42885](https://github.com/kubernetes/kubernetes/issues/42885#issuecomment-531265679),
> and [considerations for adding to the "all" category](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-cli/kubectl-conventions.md#rules-for-extending-special-resource-alias---all)
> for examples of the difficulties encountered.
>
> Categories listed in discovery may be used by clients as hints to resolve user invocations to multiple resources.
>
> Examples of built-in categories and the resources they map to include:
> * `api-extensions`
> * `apiregistration.k8s.io/v* apiservices`
> * `admissionregistration.k8s.io/v* mutatingwebhookconfigurations`
> * `admissionregistration.k8s.io/v* validatingwebhookconfigurations`
> * `admissionregistration.k8s.io/v* validatingadmissionpolicies`
> * `admissionregistration.k8s.io/v* validatingadmissionpolicybindings`
> * `apiextensions.k8s.io/v* customresourcedefinitions`
> * `all`
> * `v1 pods`
> * `v1 replicationcontrollers`
> * `v1 services`
> * `apps/v* daemonsets`
> * `apps/v* deployments`
> * `apps/v* replicasets`
> * `apps/v* statefulsets`
> * `autoscaling/v* horizontalpodautoscalers`
> * `batch/v* cronjobs`
> * `batch/v* jobs`
>
> With the above categories, and only built-in API types served, `kubectl get all` would be equivalent to
> `kubectl get pods.v1.,replicationcontrollers.v1.,services.v1.,daemonsets.v1.apps,deployments.v1.apps,replicasets.v1.apps,statefulsets.v1.apps,horizontalpodautoscalers.v2.autoscaling,cronjobs.v1.batch,jobs.v1.batch,`.

---
#### Discovering categories supported by custom resources

The `kubectl api-resources` does not support discoverying available categories: its text output does not include categories
The `kubectl api-resources` support filtering resources from a known set of categories.

The main api [discovery endpoint ` /apis`](https://kubernetes.io/docs/concepts/overview/kubernetes-api/#discovery-api) is not returning categories
```
kubectl get --raw /apis | jq
```

```json
{
"kind": "APIGroupList",
"apiVersion": "v1",
"groups": [
{
"name": "apiregistration.k8s.io",
"versions": [
{
"groupVersion": "apiregistration.k8s.io/v1",
"version": "v1"
}
],
"preferredVersion": {
"groupVersion": "apiregistration.k8s.io/v1",
"version": "v1"
}
},
...
```

Instead it requires to query invidual discovery endpoint for each resource, despite not being documented

https://kubernetes.io/docs/concepts/overview/kubernetes-api/#discovery-api
> Additional requests are needed to obtain the discovery document for each group version at /apis// (for example: /apis/rbac.authorization.k8s.io/v1alpha1), which advertises the list of resources served under a particular group version. These endpoints are used by kubectl to fetch the list of resources supported by a cluster

```bash
kubectl get --raw /apis/user.keycloak.m.crossplane.io/v1alpha1 | jq
```

```json
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "user.keycloak.m.crossplane.io/v1alpha1",
"resources": [
{
"name": "roles",
"singularName": "roles",
"namespaced": true,
"kind": "Roles",
"verbs": [
"delete",
"deletecollection",
"get",
"list",
"patch",
"create",
"update",
"watch"
],
"categories": [
"crossplane",
"managed",
"keycloak"
],
"storageVersionHash": "0Yk/XIT3FnM="
},
...
```

One common workaround for users is to not use the discovery api, and instead using the API extension api, aka CRDs

The CRD do include categories, but requires some undocumented post processing steps to extract them, such as the followings

```bash
kubectl get crds -o json | jq -r '
# Extract all CRDs
.items[]
# Get the categories array from each CRD spec
| .spec.names.categories[]?
# Remove duplicates and sort alphabetically
' | sort -u | jq -R . | jq -s .

```
produces:
```json
[
"all",
"gateway-api",
"networking"
]
```

```bash
kubectl get crds -o json | jq '
# Create an array of category-CRD pairs
[
.items[] |
# For each CRD, get its name and categories
.spec.names.categories[]? as $category |
{
category: $category,
crd: .metadata.name
}
]
# Group by category
| group_by(.category)
# Transform into desired format
| map({
category: .[0].category,
crds: map(.crd) | sort
})
# Sort by category name
| sort_by(.category)
'

```

produces
```json
[
{
"category": "all",
"crds": [
"certificates.cert-manager.io",
"gateways.gateway.networking.k8s.io"
]
},
{
"category": "gateway-api",
"crds": [
"gateways.gateway.networking.k8s.io",
"httproutes.gateway.networking.k8s.io"
]
}
]
```
---
#### Guidance for CRD authors

There are no easily discoverable guidance for CRD authors on usage of the `all` category and other freeform categories
##### All category

The following guidance is targetting kubectl contributors
https://github.com/kubernetes/community/blob/main/contributors/devel/sig-cli/kubectl-conventions.md#rules-for-extending-special-resource-alias---all
https://github.com/kubernetes/community/blob/14e2a8d7097fe4fd09b2f4d3e731922fb88c2450/contributors/devel/sig-cli/kubectl-conventions.md?plain=1#L108-L120
> Here are the rules to add a new resource to the `kubectl get all` output.
> * No cluster scoped resources
> * No namespace admin level resources (limits, quota, policy, authorization
> rules)
> * No resources that are potentially unrecoverable (secrets and pvc)
> * Resources that are considered "similar" to kubernetes/kubernetes#3 should be grouped
the same (configmaps)

Similar guidance for CRD authors would be useful in the CRD documentation

##### other categories

There are no easily discoverable guidance for CRD authors on naming categories in order to
* avoid collisions between a resource name (`clients`) and category (`clients`)
* avoid collisions among unrelated categories

See a sample discussion on https://github.com/fluxcd/flux2/issues/5947#issuecomment-4781893327
> Crossplane apparently chose _keywords_. I don't like it. I prefer _categories_:
>
> ```
> * all (lists all resources) ^ least specific
> * fluxcd (lists all flux resources) | less specific
> * fluxcd-{category} (lists all resources of a flux category) | more specific
> * {resource} (lists all resources of a flux kind) v most specific
> ```

### Kubernetes version

1.36.2

### Cloud provider

na

### OS version

_No response_

### Install tools

### Container runtime (CRI) and version (if applicable)

### Related plugins (CNI, CSI, ...) and versions (if applicable)

### Anything else we need to know?