Github backend requires full organization account access

**Describe the bug**
We want to keep a static site project in a private github repo inside our organization and use netlify-cms for content management. However, the OAuth backend requires full read/write access to all repos of that organization and a more fine-grained selection of grants is not possible.

**To Reproduce**

1. Create a new private repo inside a github organization with restrictive access (i.e. _organization settings_ → _third-party access_ → _policy = access restricted_)
2. Set up a Hugo/GatsbyJS/... site with netlify-cms in that repo [as described in the guide](https://www.netlifycms.org/docs/gatsby/)
3. Configure the `github` backend for netlify-cms
4. Deploy site on Netlify
5. Create an OAuth2 app for Github authentication [as described here](https://docs.netlify.com/visitor-access/oauth-provider-tokens/#setup-and-settings)
6. Visit the site's netlify-cms backend and click _Login with Github_

**Expected behavior**

One would expect that it's possible to restrict netlify-cms access to only the relevant repositories (just like it's possible when setting up a Netlify site from Github).

**Screenshots**

**Applicable Versions:**

- Netlify CMS version: netlify-cms-app 2.12.22 / netlify-cms-core 2.30.6
- Git provider: GitHub
- Browser/OS: Chromium Version 85.0.4183.102 (Official Build) Arch Linux (64-bit)

- Node.JS version: node v14.10.1 / npm 6.14.3

**CMS configuration**
```yml
backend:
name: github
repo: my-org/my-repo
branch: master

media_folder: static/img
public_folder: /img

collections:
- name: 'blog'
label: 'Blog'
folder: 'content/blog'
create: true
slug: 'index'
media_folder: ''
public_folder: ''
path: '{{title}}/index'
editor:
preview: false
fields:
- { label: 'Title', name: 'title', widget: 'string' }
- { label: 'Publish Date', name: 'date', widget: 'datetime' }
- { label: 'Description', name: 'description', widget: 'string' }
- { label: 'Body', name: 'body', widget: 'markdown' }
```

**Additional context**
I am aware that this issue might not be directly related to netlify-cms but to either the Netlify API or the Github OAuth API. Please let me know if I should report this upstream instead, thanks!

Our current workaround is creating a separate Github organization with a single private repo and granting the Netlify Auth Service full access to this organization.