SmartStubAdapter slower than SmartConnectNoSSL

Im writing a small internal service that uses pyvmomi.
It's a daemon that runs all the time, but don't have a continuous load.

I had issues where the session timed out and searched for at solution.

The on that looked the most promising was creating a SmartStubAdapter and VimSessionOrientedStub.
Like this:

smart_stub = SmartStubAdapter(
    host=app.config['VCENTER'],
    port=app.config['PORT'],
    sslContext=ssl._create_unverified_context(),
    connectionPoolTimeout=0
)
session_stub = VimSessionOrientedStub(
    smart_stub,
    VimSessionOrientedStub.makeUserLoginMethod(
        app.config['USER'],
        app.config['PASSWORD']
    )
)
self.si = vim.ServiceInstance('ServiceInstance', session_stub)

Instead of this:

self.si = SmartConnectNoSSL(**_connect_parameters)

It solves the problem with the session timeout, but all requests made takes about twice as long.
A particularly heavy request that lists all vms with some parameters takes 15 seconds instead of 7 seconds like it did before.

I tried reading the pyvmomi-code to see if I was using it wrong but couldnt find anything.

Has anyone else seen this issue?
Should I do something different?

Start by comparing the SmartStubAdapter/VimSessionOrientedStub setup with the SmartConnectNoSSL path shown in the issue, reproducing the reported timings on a request that lists all VMs. Done means identifying and correcting or clearly explaining the roughly 2x slowdown while retaining the session-timeout behavior.