[BUG] Presence of many Saltify'd machines can slow all cloud actions

Description

A configured saltify driver with many Saltify'd servers can dramatically slow any cloud action (such as show_instance), since an effective salt -G salt-cloud:driver:saltify grains.items runs on every action.

Going to break from the bug template and just dump what I found during a customer investigation.

I've followed the source, and it looks like this is behaving as expected for the saltify cloud driver.

• A salt-cloud action will run map_providers_parallel(): https://github.com/saltstack/salt/blob/da6819735ce055f16e326472cb1768faece72ad6/salt/cloud/__init__.py#L1446

• This will eventually run list_nodes_min() for each provider if it's available, otherwise list_nodes(): https://github.com/saltstack/salt/blob/da6819735ce055f16e326472cb1768faece72ad6/salt/cloud/__init__.py#L670-L681

• For saltify, there is no list_nodes_min(), so it runs list_nodes()...

• list_nodes() calls _list_nodes_full(): https://github.com/saltstack/salt/blob/da6819735ce055f16e326472cb1768faece72ad6/salt/cloud/clouds/saltify.py#L131

• _list_nodes_full() runs the equivalent of salt -G salt-cloud:driver:saltify grains.items: https://github.com/saltstack/salt/blob/da6819735ce055f16e326472cb1768faece72ad6/salt/cloud/clouds/saltify.py#L209

  This last bit is what takes so long in the customer environment. This will target everything that matches (according to the grains cache), and the command will wait (until timeout) for any minion for which there is no grains cache. It didn't look like there were any timeouts, though.

  It took around 21 seconds to get all of that grains data back, but only the minion names are used from that data, and then all but one don't match...

  08:34:20,891 [salt.transport.ipc                       ][DEBUG   ] Closing IPCMessageSubscriber instance
  08:34:21,032 [salt.cloud                               ][DEBUG   ] vm:foo1 in provider:saltify is not in name list:'{'foo42'}'
  08:34:21,033 [salt.cloud                               ][DEBUG   ] vm:foo2 in provider:saltify is not in name list:'{'foo42'}'
  [...]
  

• Finally, we're ready to call the action on the one matching name: https://github.com/saltstack/salt/blob/da6819735ce055f16e326472cb1768faece72ad6/salt/cloud/__init__.py#L1495-L1502

• And that just runs grains.items again against one the one VM: https://github.com/saltstack/salt/blob/da6819735ce055f16e326472cb1768faece72ad6/salt/cloud/clouds/saltify.py#L227

How to potentially make this better:

• The saltify driver should support a list_nodes_min function if possible which could do a lighter-weight retrieval of saltify'd nodes.

Versions Report

Seen in Salt 3002.2.

Additional context

ZD-6357.

Start in salt/cloud/clouds/saltify.py, especially list_nodes(), _list_nodes_full(), and the grains.items call, then compare the provider selection flow in salt/cloud/init.py. Investigate how saltify could provide a lighter list_nodes_min path without retrieving full grains for every machine. Done means cloud actions no longer perform the expensive full retrieval when only node names are needed, with relevant tests updated or added.