douglasg14b / douglasg14b/skillshare_scraper
Update Scraper to handle new review system
- Dominant language
- PHP
- Stars
- 10
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Goals:
- [ ] Collect Individual Reviews
- [ ] Load Paginated Reviews
- [ ] Load Archived Reviews
- [ ] Collect User info for reviewers
- [ ] Collect Aggregate Review Data
## Getting the Data:
**Retrieving initial data:**
```js
$.ajax({
headers: {Accept: 'application/json, text/javascript, */*; q=0.01'},
method: "GET",
url: 'https://www.skillshare.com/classes/Modern-Botanical-Folk-Art-Draw-and-Paint-Whimsical-Flowers-and-Leaves-Using-Gouache-and-Watercolor/614572280/reviews'
})
```
**Notes:**
* The Path before the SKU does not matter.
* This will load the most recent 5 reviews
* The reviews are paginated
**Iterating paginated data:**
You can do that with this endpoint:
`https://www.skillshare.com/reviews/reviewsLoadMore?sku=614572280&offset=9&collectionType=1`
The offset is as you would it expect it to be, reviews will be in descending order by date.
There are also archived reviews that can be retrieved with:
`https://www.skillshare.com/reviews/reviewsLoadMore?sku=614572280&offset=9&collectionType=2`
This works the same as the other one, but has slightly different data:
json
```json
{
"reviews":[
],
"archiveReviews":[
{
"id":"627645",
"testimonial":"Very great examples and down-to-earth explanation. Even if the class does go a little too fast at times.",
"recommended":1,
"iconStyle":"positive icon-like",
"reviewer":{
"fullName":"David Britz",
"username":"9962183",
"url":"https:\/\/www.skillshare.com\/profile\/David-Britz\/9962183",
"headline":null,
"avatar":"https:\/\/static.skillshare.com\/assets\/images\/default-profile-sm.jpg"
},
"canModify":false,
"canFlag":0,
"overallRating":null,
"levelingRating":null,
"reviewTagsPositive":[
],
"createTime":1530672057
},
{
"id":"615507",
"testimonial":"Thank you so much Alan. Your course was awesome. I learned so, so much. Your course was helpful, full of really useful information and your delivery was so nice. You are easy to understand and pleasant to listen to. \nI would highly recommend you to all. \nThanks so much. \nDanielle",
"recommended":1,
"iconStyle":"positive icon-like",
"reviewer":{
"fullName":"Danielle McCarthy",
"username":"2414065",
"url":"https:\/\/www.skillshare.com\/user\/danielleeenkhoorn86",
"headline":"Vegan and a Life Student",
"avatar":"https:\/\/static.skillshare.com\/uploads\/users\/2414065\/user-image-small.jpg?192404562"
},
"canModify":false,
"canFlag":0,
"overallRating":null,
"levelingRating":null,
"reviewTagsPositive":[
],
"createTime":1529273239
},
{
"id":"608584",
"testimonial":"Bite size tutorials of advanced Excel functions guiding you through almost everything you're employer will ever need you to do in Excel. ",
"recommended":1,
"iconStyle":"positive icon-like",
"reviewer":{
"fullName":"Joshua .",
"username":"3179584",
"url":"https:\/\/www.skillshare.com\/profile\/Joshua-.\/3179584",
"headline":null,
"avatar":"https:\/\/static.skillshare.com\/assets\/images\/default-profile-sm.jpg"
},
"canModify":false,
"canFlag":0,
"overallRating":null,
"levelingRating":null,
"reviewTagsPositive":[
],
"createTime":1528377251
},
{
"id":"433740",
"testimonial":"Alan is a very clear, concise and likable teacher. Even though this class predates Excel 2016 that I use, I still found it chock full of good tips and information. Well done!",
"recommended":1,
"iconStyle":"positive icon-like",
"reviewer":{
"fullName":"Roxane Dayton",
"username":"9209302",
"url":"https:\/\/www.skillshare.com\/profile\/Roxane-Dayton\/9209302",
"headline":null,
"avatar":"https:\/\/static.skillshare.com\/assets\/images\/default-profile-sm.jpg"
},
"canModify":false,
"canFlag":0,
"overallRating":null,
"levelingRating":null,
"reviewTagsPositive":[
],
"createTime":1504470794
},
{
"id":"414050",
"testimonial":"To say I'm impressed by Alan's teaching style is an understatement. I've learned so much from just skimming through the first two levels of his Excel \"masterclass\", I'm definitely going to have to go back through it all and *really* learn it.\nI always used to sort of write Excel off as a nerdy accountants' kind of thing, but gosh have my eyes been opened! Thanks Alan.",
"recommended":1,
"iconStyle":"positive icon-like",
"reviewer":{
"fullName":"Ian Carter",
"username":"170568",
"url":"https:\/\/www.skillshare.com\/profile\/Ian-Carter\/170568",
"headline":null,
"avatar":"https:\/\/static.skillshare.com\/assets\/images\/default-profile-sm.jpg"
},
"canModify":false,
"canFlag":0,
"overallRating":null,
"levelingRating":null,
"reviewTagsPositive":[
],
"createTime":1501110431
}
],
"aggregateReviewsData":{
"levelingRating":2,
"teacherSetLevel":false,
"tagCounts":[
{
"name":"Clarity of Instruction",
"value":7
},
{
"name":"Engaging Teacher",
"value":6
},
{
"name":"Organization of Lessons",
"value":6
}
],
"overallRatingPercentages":{
"4":{
"rating":5,
"avg":63
},
"3":{
"rating":3,
"avg":38
}
}
},
"numReviews":8,
"numArchiveReviews":39,
"loadMoreReviews":true,
"loadMoreArchiveReviews":true
}
```
## Notes
The `reviewTagsPositie` refers to the `What did you like about this class?` section of the review. This is an array of the items selected from 1-6 in the following order:
1. `Engaging Teacher`
2. `Organization of Lessons`
3. `Clarity of Instruction`
4. `Helpful Examples`
5. `Actionable Steps`
6. `Audio & Video Quality`
## JSON Structure
Response Example
```json
{
"reviews":[
{
"id":764784,
"testimonial":"Handy fundamentals, my spouse is getting a DSLR camera so this should be helpful.",
"recommended":0,
"iconStyle":"negative icon-dislike",
"reviewer":{
"fullName":"Ghectavk Ghectavk",
"username":4420607,
"url":"https://www.skillshare.com/profile/Ghectavk-Ghectavk/4420607",
"headline":null,
"avatar":"https://static.skillshare.com/assets/images/default-profile-sm.jpg"
},
"canModify":true,
"canFlag":0,
"overallRating":3,
"levelingRating":2,
"reviewTagsPositive":[
1,
6
],
"createTime":1549595266
},
{
"id":763534,
"testimonial":"I think showing some bad examples of pictures would be really helpful because while I think the examples shown are really nice to look at they don't exactly show what you should avoid when taking a picture.",
"recommended":0,
"iconStyle":"negative icon-dislike",
"reviewer":{
"fullName":"Isabelle Gentry",
"username":77516,
"url":"https://www.skillshare.com/profile/Isabelle-Gentry/77516",
"headline":"Just here to make nice things.",
"avatar":"https://static.skillshare.com/uploads/users/77516/user-image-small.png?1157890235"
},
"canModify":false,
"canFlag":0,
"overallRating":4,
"levelingRating":1,
"reviewTagsPositive":[
1,
2,
3,
6
],
"createTime":1549430199
},
{
"id":761149,
"testimonial":"I wish I started here, this has everything in one video that I pieced together watching tons of other Skillshare teachers. would have made a faster improvement if I started here. I'm about to watch everything else he has to offer. thanks soo much for making things soo understandable especially like that you show the meter along the way.",
"recommended":0,
"iconStyle":"negative icon-dislike",
"reviewer":{
"fullName":"David Jackson",
"username":8576655,
"url":"https://www.skillshare.com/profile/David-Jackson/8576655",
"headline":null,
"avatar":"https://graph.facebook.com/10157006700691349/picture"
},
"canModify":false,
"canFlag":0,
"overallRating":4,
"levelingRating":4,
"reviewTagsPositive":[
1,
2,
3,
4,
5,
6
],
"createTime":1549121011
},
{
"id":760633,
"testimonial":"I've been dabbling in digital photography a little and had trouble reading grasping the triangle. iso, shutter and f/stop. This class broke it down great and made it very comprehendible and easy to understand for me a least. Thanks",
"recommended":0,
"iconStyle":"negative icon-dislike",
"reviewer":{
"fullName":"Laurel Rahn",
"username":6365806,
"url":"https://www.skillshare.com/profile/Laurel-Rahn/6365806",
"headline":null,
"avatar":"https://static.skillshare.com/assets/images/default-profile-sm.jpg"
},
"canModify":false,
"canFlag":0,
"overallRating":4,
"levelingRating":1,
"reviewTagsPositive":[
1,
2,
3,
4
],
"createTime":1549042062
},
{
"id":760513,
"testimonial":"Great class!Justin has a very clear, simple and straight to the point way of teaching. Really recommended. If I was living in NY I would love taking classes with him. Ciao from Italy ;)",
"recommended":0,
"iconStyle":"negative icon-dislike",
"reviewer":{
"fullName":"ELISA CHESSA",
"username":9691003,
"url":"https://www.skillshare.com/profile/ELISA-CHESSA/9691003",
"headline":null,
"avatar":"https://static.skillshare.com/uploads/users/9691003/user-image-small.jpg?273958410"
},
"canModify":false,
"canFlag":0,
"overallRating":4,
"levelingRating":1,
"reviewTagsPositive":[
1,
3,
4
],
"createTime":1549028785
}
],
"archiveReviews":[
],
"aggregateReviewsData":{
"levelingRating":1,
"teacherSetLevel":false,
"tagCounts":[
{
"name":"Clarity of Instruction",
"value":228
},
{
"name":"Helpful Examples",
"value":227
},
{
"name":"Engaging Teacher",
"value":220
}
],
"overallRatingPercentages":{
"3":{
"rating":83,
"avg":31
},
"4":{
"rating":184,
"avg":69
},
"2":{
"rating":1,
"avg":0
}
}
},
"numNewReviews":268,
"numArchiveReviews":2023,
"numArchiveRecommendations":2021,
"reviewsSortOrder":{
"tabs":[
{
"name":"highest-rated",
"label":"Highest Rated",
"link":"?reviewsSort=highest-rated",
"active":false
},
{
"name":"lowest-rated",
"label":"Lowest Rated",
"link":"?reviewsSort=lowest-rated",
"active":false
},
{
"name":"most-recent",
"label":"Most Recent",
"link":"?reviewsSort=most-recent",
"active":true
}
]
},
"currentSort":"most-recent",
"expectationsSort":"",
"loadMoreUrl":"https://www.skillshare.com/reviews/reviewsLoadMore?sku=1111783378",
"loadMoreReviews":true,
"loadMoreArchiveReviews":true,
"currentUserReviewData":{
"id":[snip],
"status":4,
"recommended":null,
"testimonial":"[snip]",
"criticism":null,
"create_time":"[snip]",
"update_time":"[snip]",
"class_id":[snip],
"teacher_uid":2451539,
"reviewer_id":[snip],
"follow_up":null,
"is_reviewer_deleted":0,
"is_deleted":0,
"would_recommend":null,
"parent_class_id":[snip],
"flag_status":0,
"overall_rating":[snip],
"leveling_rating":[snip],
"tags":[
[snip]
]
},
"currentUserHasReviewed":true,
"reviewsEmpty":false,
"canReview":true
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the scraper entry point that retrieves the initial reviews response, then compare it with the reviewsLoadMore endpoints for collectionType 1 and 2. Use the response examples to identify individual, reviewer, archived, and aggregate data; done means all listed review data is collected across pagination and archives.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100