Directory API: Not Authorized to access this resource/api

I created a project in the Google Cloud Platform and went ahead and activated the Admin SDK & Gmail APIs. I then created a domain-wide service account in the project, downloaded the JSON key file and gave it authorization to the following scopes in the Admin Console:
```
https://www.googleapis.com/auth/admin.directory.user
https://www.googleapis.com/auth/admin.directory.group
https://www.googleapis.com/auth/admin.directory.group.member
https://www.googleapis.com/auth/gmail.settings.basic
```

Using the JSON file, I can create a JWT auth object and access the Gmail API just fine. However, I keep getting `403: Not Authorized to access this resource/api` when attempting to use any resource on the Directory API.

In the API & Services Dashboard on the Google Cloud Platform, I can see the requests coming in and being denied but I can't think of any reason why this would not work properly.

Here is the code I am using (of course using a different domain than mydomain.com):
```js
const useDirectory = async keyFile => {
const auth = new google.auth.GoogleAuth({
keyFile,
scopes: [
'https://www.googleapis.com/auth/admin.directory.user',
'https://www.googleapis.com/auth/admin.directory.group'
'https://www.googleapis.com/auth/admin.directory.group.member'
]
});

return google.admin({
version: "directory_v1",
auth: await auth.getClient()
});
};

const token = path.resolve("./token.json");
if (!fs.existsSync(token)) {
throw new Error("Could not find token.json for authentication.");
}

const directory = await useDirectory(token);
const users = await directory.groups
.list({ domain: "mydomain.com" })
.catch(console.error);

```

#### Environment details

- OS: macOS Mojave 10.14.6
- Node.js version: 10.16.3
- npm version: 6.11.3
- `google-auth-library` version: 5.2.0

#### Steps to reproduce

Too many steps based on the description of the issue.