Extract Comments for Issues, Merge Requests and Epics

In GitLab by @iroussos on Nov 12, 2019, 07:27

We are currently not extracting the Comments on Issues and Merge Requests.

All comments can be fetched by using the notes API end point:

1. List project issue notes
2. List merge request notes
3. List epic notes (should be enabled only if "ultimate_license": true)

Similarly to fetching the Commits for a Merge Request, this operation requires one API call per Merge Request (or Issue / Epic), so going through that flow can slow down considerably the end-to-end extraction time. For example, in a project like gitlab-org/gitlab-foss, this would result to an additional ~90K more API calls.

The approach should be similar to what we did with fetching the Commits for a Merge Request, with a configuration parameter similar to fetch_merge_request_commits that will allow users to manually set this option for projects they know can handle that type of N+1 querying.

Useful Resources when working with Gitlab's API:

1. Gitlab API Documentation
2. Gitlab API entities
3. Entity Schemas

Start by comparing the existing merge-request commit extraction flow and its fetch_merge_request_commits configuration with GitLab's issue, merge request, and epic notes API endpoints. Check the linked GitLab API documentation, entity definitions, and schema resources. Done means comments can be optionally fetched for each resource, epic notes respect ultimate_license, and the extra N+1 requests are disabled unless configured.