AbrahamSanders / AbrahamSanders/SIMIE

[Dev] Support downloading and parsing books from smashwords.com

Offen
#3 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
enhancement
Vorherrschende Sprache
Python
Sterne
5
Forks
0
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

[smashwords.com](https://www.smashwords.com/) was used as the source of the original [BookCorpus](https://yknzhu.wixsite.com/mbweb) dataset, built for the 2015 paper [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://arxiv.org/abs/1506.06724).

We should support smashwords as an alternate source of books, since it can provide more modern works than those in project Gutenberg. Dialogs and narratives written in a modern style are absolutely necessary to train a model that will work well with the way people speak and write today.

This likely involves:

1. Implementing a downloader for [smashwords.com](https://www.smashwords.com/)
2. Implementing an adapter, if necessary, to format the downloaded texts in the way that [gutenberg-dialog](https://github.com/AbrahamSanders/gutenberg-dialog)'s pipeline expects.

https://github.com/soskek/bookcorpus may be a good starting point, as it implements a crawler for smashwords.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.