[BUG] cloud metadata grain processor does not json-decode metadata values

Description
The cloud metadata grain processor does not json-decode the metadata values.

This causes grain targeting of cloud metadata (for instance, the AWS account ID or EC2 instance ID from http://169.254.169.254/latest/dynamic/instance-identity/document, much more difficult, since the grain contains a JSON object rather than being parsed into specific fields.

Setup

• launch an AWS ec2 instance (or any other cloud instance with metadata provided at 169.254.169.254, as supported by the grains/metadata.py grain handler; the examples use AWS)
• install latest salt and salt-minion (in this case, via bootstrap)
• set metadata_server_grains: true in /etc/salt/minion and restart minion
• refresh grains: salt-call saltutil.refresh_grains

Steps to Reproduce the behavior

Fetch and review the dynamic:instance-identity:document grain, which contains EC2 instance information such as account ID and instance ID:

sudo salt-call grains.get dynamic:instance-identity:document --out=json

and observe that the returned grain is a string containing a json blob:

{
    "local": "{\n  \"accountId\" : \"XXXXXXXXXXXX\",\n  \"architecture\" : \"x86_64\",\n  \"availabilityZone\" : \"us-east-2a\",\n  \"billingProducts\" : null,\n  \"devpayProductCodes\" : null,\n  \"marketplaceProductCodes\" : null,\n  \"imageId\" : \"ami-XXXXXXXXXXXXXXXXX\",\n  \"instanceId\" : \"i-XXXXXXXXXXXXXXXXX\",\n  \"instanceType\" : \"t3.xlarge\",\n  \"kernelId\" : null,\n  \"pendingTime\" : \"2023-05-02T19:10:58Z\",\n  \"privateIp\" : \"172.17.80.65\",\n  \"ramdiskId\" : null,\n  \"region\" : \"us-east-2\",\n  \"version\" : \"2017-09-30\"\n}"
}

Expected behavior

Returned grain is a dictionary containing the relevant data. For example, using a modified version of this grains processor that performs the JSON decoding, I get the following back instead:

{
    "local": {
        "accountId": "XXXXXXXXXXXX",
        "architecture": "x86_64",
        "availabilityZone": "us-east-2a",
        "billingProducts": null,
        "devpayProductCodes": null,
        "marketplaceProductCodes": null,
        "imageId": "ami-XXXXXXXXXXXXXXXXX",
        "instanceId": "i-XXXXXXXXXXXXXXXXX",
        "instanceType": "t3.xlarge",
        "kernelId": null,
        "pendingTime": "2023-05-02T19:10:58Z",
        "privateIp": "172.17.80.65",
        "ramdiskId": null,
        "region": "us-east-2",
        "version": "2017-09-30"
    }
}

which allows for grain targeting such as dynamic:instance-identity:document:accountId:XXXXXXXXXXXX to target systems belonging to a specific AWS account.

Versions Report

$ salt-minion --versions-report
Salt Version:
          Salt: 3006.1

Python Version:
        Python: 3.10.11 (main, May  5 2023, 02:31:54) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
        relenv: 0.12.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: centos 7.9.2009 Core
        locale: utf-8
       machine: x86_64
       release: 3.10.0-1127.el7.x86_64
        system: Linux
       version: CentOS Linux 7.9.2009 Core

Additional context

This appears to possibly be a py3-conversion issue, since there is a codepath for performing the conversion prior to the py3 conversion:

https://github.com/saltstack/salt/blob/e312efb5e77db40d4574b24bcb7a4aefcda05cf9/salt/grains/metadata.py#L70-L76

but in the first round of py3 conversion (with six) it only decodes for six.binary_type (str in Py2, bytes in py3):

https://github.com/saltstack/salt/blob/396f7906e33123a2751fd3544674380f895f3d21/salt/grains/metadata.py#L77-L83

and then when six was removed, it explicitly only decodes for bytes:

https://github.com/saltstack/salt/blob/d9b50659b2be467f31d59245e70202fbabc83092/salt/grains/metadata.py#L74-L82

which fails to decode, for instance, the AWS metadata path http://169.254.169.254/latest/dynamic/instance-identity/document, since that comes into this function as str, not bytes.

Potential solution, with compatibility concern

I have worked around this in a local copy of metadata.py by changing https://github.com/saltstack/salt/blob/d9b50659b2be467f31d59245e70202fbabc83092/salt/grains/metadata.py#L74 to:

            if isinstance(retdata, bytes) or isinstance(retdata, str):

which maintains, as far as I can tell, the original intent.

This does present a compatibility problem: if someone has worked around this issue by reprocessing the metadata ingested by the current metadata.py, this change - which changes behavior that has been in place for something like five years! - would break that. However, given the degree of impact this has on functionality, I feel it's probably worthwhile to approach anyway.

One possible path would be to deprecate the existing metadata.py and create a new cloud_metadata.py, which could do the above-described json decoding and also insert its grains in a namespace, such as cloud_metadata, by changing the metadata() function to:

def metadata():
    return {'cloud_metadata': _search()}

and changing the option used from metadata_server_grains to something else, such as:

    if __opts__.get("cloud_metadata_server_grains", False) is False:
        return False

This would allow users of the existing functionality an opportunity to migrate or adopt the old code, while providing a path forward for the functionality which appears to be both originally intended and of more general utility.

The affected entry point is salt/grains/metadata.py; start by reading the metadata grain processor and its JSON-decoding path. Reproduce with metadata_server_grains enabled and salt-call grains.get dynamic:instance-identity:document --out=json. Done means JSON metadata values are returned as dictionaries for usable grain targeting while addressing the reported compatibility concern.