[BUG] network.system state messes up DNS search domain

Description
When applying a new hostname, execution of the network.system state messes up the DNS search domain in /etc/resolv.conf by putting it in like search ['my.domain.com']. And the new hostname doesn't even get applied.

Very annoying but not functionally disrupting the minion is that salt sometimes inserts an empty line after every line in the existing /etc/resolv.conf. It looks like this happens only for larger /etc/resolv.conf files, but I haven't found the trigger for when it happens with the whole file and when it happens only with the comments.

Setup

Here is a minimal test state to trigger this bug:

# test.sls
system:
  network.system:
    - enabled: True
    - hostname: minion-01.test.my.domain.com
    - apply_hostname: True
    - retain_settings: True

Please be as specific as possible and give set-up details.

• on-prem machine (on-prem hypervisor and naked Raspberrypi)
• VM (VMware ESXi guest, both master and minion)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD

Issue is reproducible on minions running Debian 11 as VMware guest, Ubunut 20.04 as VMware guest, and Ubuntu 20.04 on a Raspberrypi. Haven't tested on other variations.

Master and all minions are on salt v3004.1.

Steps to Reproduce the behavior

Before applying the state (handcrafted/manually fixed /etc/resolv.conf file on the minion):

# /etc/resolv.conf
# There will be an empty line below this comment once salt has processed network.system state
search test.my.domain.com
nameserver 10.x.y.z

Next, apply a minimal state like this:

# test.sls
system:
  network.system:
    - enabled: True
    - hostname: minion-01.test.my.domain.com
    - apply_hostname: True
    - retain_settings: True

# salt 'minion-01' state.sls test

The resulting file on the minion looks like this. Note the invalid search domain specification.

# /etc/resolv.conf
# There will be an empty line below this comment once salt has processed network.system state

search ['test.my.domain.com']
nameserver 10.x.y.z

Sometimes I even end up with:

search ["['test.my.domain.com']"]

But I've given up trying to reproduce this even faultier outcome.

Calling the following on the minion directly produces the same wrong outcome:

sudo salt-call ip.build_network_settings enabled=True hostname=minion-01.test.my.domain.com apply_hostname=True
local:
    - NETWORKING=yes
    - HOSTNAME=minion-01
    - DOMAIN=test.my.domain.com
    - SEARCH=['test.my.domain.com']

The icing on the top is that with all the DNS search domain mess, the changed hostname doesn't even get applied (in case there is a hostname change).

Expected behavior

Generate/modify search domain setting in /etc/resolv.conf without any braces and/or single/double quotation marks.

# ...
search test.my.domain.com
# ...

Versions Report

Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: Not Installed
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-13-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

Additional context

This might be related to other bugs in network.system: #32983 and #6922

Start with the network.system state and the ip.build_network_settings command shown in the report, then reproduce the issue using the provided test.sls and /etc/resolv.conf examples. Done means the search domain is written without brackets or quotes, the hostname is applied, and existing resolver content is not unexpectedly spaced.