nelsonic / nelsonic/github-scraper

Scraping of issues list is broken?

Open
#107 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted
Dominant language
JavaScript
Stars
456
Forks
97
PR merge metrics
No merged PRs in 30d

Description

I'm trying to use the scraper to get all the issues in a bunch of repositories. However, the scraper appears to behave differently from what it is described in the documentation.
I have copied the example code in a file:

var gs  = require('github-scraper');
var url = '/dwyl/tudo/issues';
gs(url, function (err, data) {
  console.log(data); // use the data how ever you like
});

and executed it. My output appears to return repository stats:

- - - GitHub Scraper >> /dwyl/tudo/issues/ >> repo  - - -  
{
  url: '/dwyl/tudo/issues/',
  type: 'repo',
  description: '',
  website: '',
  tags: '',
  watchers: 27,
  stars: 52,
  forks: 8,
  commits: NaN,
  branches: NaN,
  releases: 34,
  langs: []
}

instead of issues list as expected (output given in the example):

{ entries:
   [
     {
       url: '/dwyl/tudo/issues/46',
       title: 'discuss components',
       created: '2015-07-21T15:34:22Z',
       author: 'benjaminlees',
       comments: 3,
       assignee: 'izaakrogan',
       milestone: 'I don\'t know what I\'m doing',
       labels: [ 'enhancement', 'help wanted', 'question' ]
     },
     {
       url: '/dwyl/tudo/issues/45',
       title: 'Create riot components from HTML structure files',
       created: '2015-07-21T15:24:58Z',
       author: 'msmichellegar',
       comments: 2,
       assignee: 'msmichellegar',
       labels: [ 'question' ]
     }
  ], // truncated for brevity
  open: 30,
  closed: 20,
  next: '/dwyl/tudo/issues?page=2&q=is%3Aissue+is%3Aopen',
  url: '/dwyl/tudo/issues'
}

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the documented scraper example with /dwyl/tudo/issues and compare the returned repository stats with the expected issues-list structure in this report. Trace how the scraper classifies that URL and verify the result against the documented entries, counts, pagination link, and URL; done means the example returns an issues list rather than repository statistics.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.