[BUG] OpenSSL error after update to 3006.2

Description
In my company we use an external pillar (pki) which depends on pyOpenSSL module.
It worked until i've upgraded salt-master to 3006.2, now i get this error each time i try to highstate from minion :

Aug 21 15:18:48 salt-master[1440212] salt.loaded.ext.pillar.pki: [PKI] [('x509 certificate routines', '', 'duplicate attribute')]
Aug 21 15:18:48 salt-master[1440212] salt.loaded.ext.pillar.pki: Traceback (most recent call last):
  File "/data/salt/modules/pillar/pki.py", line 1009, in ext_pillar
    p12.export(minion_id.encode("utf-8"))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/OpenSSL/crypto.py", line 2795, in export
    _raise_current_error()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/OpenSSL/_util.py", line 57, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.crypto.Error: [('x509 certificate routines', '', 'duplicate attribute')]

It seems that 3006.2 ships with a new OpenSSL version (3.1.2) and that we hit this issue : https://github.com/openssl/openssl/pull/21675

Setup
Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Expected behavior
External pillar execution shoud execute without error as in 3006.1.

Versions Report

Salt Version:
          Salt: 3006.2
 
Python Version:
        Python: 3.10.12 (main, Aug  3 2023, 21:47:10) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 20.04.6 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-153-generic
        system: Linux
       version: Ubuntu 20.04.6 focal