jmcarp / jmcarp/robobrowser

Simple AJAX request

Open
#54 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.7k
Forks
336
PR merge metrics
No merged PRs in 30d

Description

Hello. I'm try to emulate simple ajax request.

Scheme:
![image](https://cloud.githubusercontent.com/assets/4608879/10523283/3d6b93b4-7382-11e5-8894-179e683b5db3.png)

It' must return json:
`{phone: "8 xxx xxx-xx-xx"}`

But return `None` for me.
I'm suggest that the cause - cookies. Can someone help me with it?

```
# -*- coding: utf-8 -*-
import requests
import os, re, json, csv, sys
from robobrowser import RoboBrowser

class Aggregator(object):

def __init__(self, config):
main_url, output_file = [config.get(k) for k in sorted(config.keys())]
self.main_url = main_url
self.output_file = output_file

def start_process(self):

work_url = "https://m.avito.ru/sankt-peterburg/predlozheniya_uslug/almaznoe_burenie_almaznaya_rezka_usilenie_79225740"

session = requests.Session()
session.headers.update({
':host': 'm.avito.ru',
':method': 'GET',
':path': '/sankt-peterburg/predlozheniya_uslug/almaznoe_burenie_almaznaya_rezka_usilenie_79225740',
':scheme': 'https',
':version': 'HTTP/1.1',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'accept-encoding': 'gzip, deflate, sdch',
'accept-language': 'ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4',
'cache-control': 'no-cache',
'pragma': 'no-cache',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36',
'x-compress': 'null',
})
browser = RoboBrowser(session=session, history=True, parser='lxml')

browser.open(work_url)

phone_link = browser.find('a', {"class": "action-show-number"}).attrs['href'] + '?async'
browser.session.headers[':path'] = phone_link
browser.session.headers['accept'] = 'application/json, text/javascript, */*; q=0.01'
browser.session.headers['referer'] = work_url
browser.session.headers['upgrade-insecure-requests'] = ''
browser.session.headers['x-requested-with'] = 'XMLHttpRequest'
phone = browser.open(self.main_url + phone_link)

print(self.main_url + phone_link, phone)
print(browser.session.headers)
print(browser.session.cookies)

if __name__ == '__main__':
settings = { 'main_url': 'https://m.avito.ru', 'output_file': 'output.csv' }
aggregator = Aggregator(settings)
aggregator.start_process()
```

Contributor guide

Open the contributing guide

Research direction

The issue provides a Python script using requests and RoboBrowser but names no repository file, test, or entry point. Start by reproducing the supplied request flow and inspect the returned value, session headers, and cookies. Done means determining whether RoboBrowser is responsible and documenting a reproducible fix or clear limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.