turicas / turicas/rows

[HTML] Add option to extract links

Open
#166 0 comments 0 reactions 1 assignee View on GitHub

@turicas is already working on this.

Since Jul 17, 2016.

enhancement plugin
Dominant language
Python
Stars
886
Forks
137
PR merge metrics
No merged PRs in 30d

Description

When creating crawlers, sometimes the information we want are links and they are inside some table cells.
It would be useful to extract them quickly (using the library or its CLI), like this:

rows print --table-index=1 --fields=brand,model,device_techdata.link 'https://wiki.openwrt.org/toh/start'

I'm using .link above to specify I want the first link inside the column device_techdata (in this case the field device_techdata should not be imported, only the link inside it). It could be extended also to .links for cases when the cell has more than one link.

I'm currently implementing this outside the library (but probably it could be more optimized if inside) with the following code:

from __future__ import unicode_literals

from io import BytesIO

import requests
import rows

from rows.plugins.html import extract_links


def extract_field_link(field_name):
    def func(row):
        data = getattr(row, field_name)
        if data is None:
            return None
        else:
            links = extract_links(data)
            if not links:
                return None
            else:
                return links[0]

    return func

def extract_field_text(field_name):
    def func(row):
        data = getattr(row, field_name)
        if data is None:
            return None
        else:
            return extract_text(data)

    return func

def import_with_links(url, fields, index=0):
    html = requests.get(url).content

    import_fields = []
    for field in fields:
        field_name = field.split('.')[0]
        if field_name not in import_fields:
            import_fields.append(field_name)

    table = rows.import_from_html(
            BytesIO(html),
            index=index,
            preserve_html=True,
            import_fields=import_fields)

    for field in fields:
        if field.endswith('.link'):
            field = field.split('.')[0]
            table['{}_link'.format(field)] = map(extract_field_link(field),
                                                 table)
            del table[field]
        else:
            table[field] = map(extract_field_text(field), table)

    return table


if __name__ == '__main__':
    table = import_with_links('https://wiki.openwrt.org/toh/start',
                              ['brand', 'model', 'device_techdata.link'],
                              index=1)
    print table[1]
    print rows.export_to_txt(table)

The output will be:

Row(brand=u'3Com',
    model=u'3CRWER100-75',
    device_techdata_link=u'https://wiki.openwrt.org/toh/hwdata/3com/3com_3crwer100-75')

+-------------+-----------------------+--------------------------------------------------------------------+
|    brand    |         model         |                        device_techdata_link                        |
+-------------+-----------------------+--------------------------------------------------------------------+
|             |                       |                                                                    |
|        3Com |          3CRWER100-75 |         https://wiki.openwrt.org/toh/hwdata/3com/3com_3crwer100-75 |
|        3Com |          3CRWER200-75 |         https://wiki.openwrt.org/toh/hwdata/3com/3com_3crwer200-75 |
|        3Com |         3CRWDR100A-72 |        https://wiki.openwrt.org/toh/hwdata/3com/3com_3crwdr100a-72 |
|  4G Systems | AccessCube (MeshCube) | https://wiki.openwrt.org/toh/hwdata/4gsystems/4gsystems_accesscube |
|      7Links |               PX-4885 |          https://wiki.openwrt.org/toh/hwdata/7links/7links_px-4885 |
|    8devices |           Carambola 2 |   https://wiki.openwrt.org/toh/hwdata/8devices/8devices_carambola2 |
|    8devices |           Carambola 1 |   https://wiki.openwrt.org/toh/hwdata/8devices/8devices_carambola1 |
|         ADB |                 1000g |                  https://wiki.openwrt.org/toh/hwdata/adb/adb_1000g |
|         ADB |               AV4202N |                https://wiki.openwrt.org/toh/hwdata/adb/adb_av4202n |
|      D-Link |          DSL-2542B/3B |      https://wiki.openwrt.org/toh/hwdata/d-link/d-link_dsl-2542b3b |
|     Linksys |             WRT1900AC |   https://wiki.openwrt.org/toh/hwdata/linksys/linksys_wrt1900ac_v2 |
| QEMU (i386) |      qemu-system-i386 |          https://wiki.openwrt.org/toh/hwdata/qemu/qemu_qemu_0125_1 |
|     TP-Link |            TL-WA701ND |     https://wiki.openwrt.org/toh/hwdata/tp-link/tp-link_tl-wa701nd |
[...]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.