[HTML] Add option to extract links
@turicas is already working on this.
Since Jul 17, 2016.
- Dominant language
- Python
- Stars
- 886
- Forks
- 137
- PR merge metrics
- No merged PRs in 30d
Description
When creating crawlers, sometimes the information we want are links and they are inside some table cells.
It would be useful to extract them quickly (using the library or its CLI), like this:
rows print --table-index=1 --fields=brand,model,device_techdata.link 'https://wiki.openwrt.org/toh/start'
I'm using .link above to specify I want the first link inside the column device_techdata (in this case the field device_techdata should not be imported, only the link inside it). It could be extended also to .links for cases when the cell has more than one link.
I'm currently implementing this outside the library (but probably it could be more optimized if inside) with the following code:
from __future__ import unicode_literals
from io import BytesIO
import requests
import rows
from rows.plugins.html import extract_links
def extract_field_link(field_name):
def func(row):
data = getattr(row, field_name)
if data is None:
return None
else:
links = extract_links(data)
if not links:
return None
else:
return links[0]
return func
def extract_field_text(field_name):
def func(row):
data = getattr(row, field_name)
if data is None:
return None
else:
return extract_text(data)
return func
def import_with_links(url, fields, index=0):
html = requests.get(url).content
import_fields = []
for field in fields:
field_name = field.split('.')[0]
if field_name not in import_fields:
import_fields.append(field_name)
table = rows.import_from_html(
BytesIO(html),
index=index,
preserve_html=True,
import_fields=import_fields)
for field in fields:
if field.endswith('.link'):
field = field.split('.')[0]
table['{}_link'.format(field)] = map(extract_field_link(field),
table)
del table[field]
else:
table[field] = map(extract_field_text(field), table)
return table
if __name__ == '__main__':
table = import_with_links('https://wiki.openwrt.org/toh/start',
['brand', 'model', 'device_techdata.link'],
index=1)
print table[1]
print rows.export_to_txt(table)
The output will be:
Row(brand=u'3Com',
model=u'3CRWER100-75',
device_techdata_link=u'https://wiki.openwrt.org/toh/hwdata/3com/3com_3crwer100-75')
+-------------+-----------------------+--------------------------------------------------------------------+
| brand | model | device_techdata_link |
+-------------+-----------------------+--------------------------------------------------------------------+
| | | |
| 3Com | 3CRWER100-75 | https://wiki.openwrt.org/toh/hwdata/3com/3com_3crwer100-75 |
| 3Com | 3CRWER200-75 | https://wiki.openwrt.org/toh/hwdata/3com/3com_3crwer200-75 |
| 3Com | 3CRWDR100A-72 | https://wiki.openwrt.org/toh/hwdata/3com/3com_3crwdr100a-72 |
| 4G Systems | AccessCube (MeshCube) | https://wiki.openwrt.org/toh/hwdata/4gsystems/4gsystems_accesscube |
| 7Links | PX-4885 | https://wiki.openwrt.org/toh/hwdata/7links/7links_px-4885 |
| 8devices | Carambola 2 | https://wiki.openwrt.org/toh/hwdata/8devices/8devices_carambola2 |
| 8devices | Carambola 1 | https://wiki.openwrt.org/toh/hwdata/8devices/8devices_carambola1 |
| ADB | 1000g | https://wiki.openwrt.org/toh/hwdata/adb/adb_1000g |
| ADB | AV4202N | https://wiki.openwrt.org/toh/hwdata/adb/adb_av4202n |
| D-Link | DSL-2542B/3B | https://wiki.openwrt.org/toh/hwdata/d-link/d-link_dsl-2542b3b |
| Linksys | WRT1900AC | https://wiki.openwrt.org/toh/hwdata/linksys/linksys_wrt1900ac_v2 |
| QEMU (i386) | qemu-system-i386 | https://wiki.openwrt.org/toh/hwdata/qemu/qemu_qemu_0125_1 |
| TP-Link | TL-WA701ND | https://wiki.openwrt.org/toh/hwdata/tp-link/tp-link_tl-wa701nd |
[...]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.