Unable to list files with service account auth for GDriveFilesystem

Hello, I'm using the latest version of the library `PyDrive2[fsspec]==1.17.0` and I'm trying to authenticate a service account (the SA email is shared with the drive I wish to test and the keys were generated, I doubt if this order matters).

I followed the steps provided in [the docs](https://docs.iterative.ai/PyDrive2/fsspec/#service-account) for the jsonkey file as a string. Its authenticated without errors however I'm unable to list files from the shared drive
1. GDrive API is hit
1. Listing of files with GDriveFilesystem returns empty (screenshot while debugging)

Digging further, I followed steps mentioned in [this comment](https://github.com/iterative/PyDrive2/issues/21#issuecomment-929186260) and was able to list files with the `GDrive` object however. In fact I noticed the `GDrive` object being created [here](https://github.com/iterative/PyDrive2/blob/4d83d04948f16804026eaf4449ac02c75ba907a4/pydrive2/fs/spec.py#L234) and the files get listed (while debugging) when I used
```
file_list = self.client.ListFile().GetList()
```

Code snippet:
```python
from oauth2client.service_account import ServiceAccountCredentials
from pydrive2.auth import GoogleAuth
from pydrive2.fs import GDriveFileSystem
from pydrive2.drive import GoogleDrive

def _init_with_service_key(self, service_key_settings: str) -> GDriveFileSystem:
"""Creates a GDriveFileSystem with a service account key."""
# Attempt from issue #21
# scope = ["https://www.googleapis.com/auth/drive"]
# gauth = GoogleAuth()
# gauth.auth_method = 'service'
# gauth.credentials = ServiceAccountCredentials.from_json_keyfile_dict(service_key_settings, scopes=scope)
# drive = GoogleDrive(gauth)

# about = drive.GetAbout()
# file_list = drive.ListFile().GetList()
# gdrive = GDriveFileSystem(
# path="root",
# google_auth=gauth,
# use_service_account=True
# )

gdrive = GDriveFileSystem(
path="root",
use_service_account=True,
client_json=service_key_settings,
skip_instance_cache=True,
use_listings_cache=False
)
return gdrive
```

Any ideas on how I can proceed? Thanks in advance! :)