Directory API: Not Authorized to access this resource/api

I created a project in the Google Cloud Platform and went ahead and activated the Admin SDK & Gmail APIs. I then created a domain-wide service account in the project, downloaded the JSON key file and gave it authorization to the following scopes in the Admin Console:
```
https://www.googleapis.com/auth/admin.directory.user
https://www.googleapis.com/auth/admin.directory.group
https://www.googleapis.com/auth/admin.directory.group.member
https://www.googleapis.com/auth/gmail.settings.basic
```

Using the JSON file, I can create a JWT auth object and access the Gmail API just fine. However, I keep getting `403: Not Authorized to access this resource/api` when attempting to use any resource on the Directory API.

In the API & Services Dashboard on the Google Cloud Platform, I can see the requests coming in and being denied but I can't think of any reason why this would not work properly.

Here is the code I am using (of course using a different domain than mydomain.com):
```js
const useDirectory = async keyFile => {
const auth = new google.auth.GoogleAuth({
keyFile,
scopes: [
'https://www.googleapis.com/auth/admin.directory.user',
'https://www.googleapis.com/auth/admin.directory.group'
'https://www.googleapis.com/auth/admin.directory.group.member'
]
});

return google.admin({
version: "directory_v1",
auth: await auth.getClient()
});
};

const token = path.resolve("./token.json");
if (!fs.existsSync(token)) {
throw new Error("Could not find token.json for authentication.");
}

const directory = await useDirectory(token);
const users = await directory.groups
.list({ domain: "mydomain.com" })
.catch(console.error);

```

#### Environment details

- OS: macOS Mojave 10.14.6
- Node.js version: 10.16.3
- npm version: 6.11.3
- `google-auth-library` version: 5.2.0

#### Steps to reproduce

Too many steps based on the description of the issue.

Inizia dall’helper useDirectory e dalla chiamata directory.groups.list, quindi confronta la delega dell’account di servizio e l’autorizzazione di Admin Console con gli Directory scopes richiesti. Riproduci il 403 usando i dettagli dell’ambiente forniti e determina se il problema è legato alla libreria o a una configurazione esterna; il lavoro è concluso quando la causa è documentata e una richiesta Directory ha esito positivo oppure il difetto della libreria è stato isolato.

Scritto dal modello di indicizzazione a partire dal testo della issue.