AbrahamSanders / AbrahamSanders/SIMIE

[Dev] Support downloading and parsing books from smashwords.com

Abierto
#3 0 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
5
Forks
0
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

[smashwords.com](https://www.smashwords.com/) was used as the source of the original [BookCorpus](https://yknzhu.wixsite.com/mbweb) dataset, built for the 2015 paper [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://arxiv.org/abs/1506.06724).

We should support smashwords as an alternate source of books, since it can provide more modern works than those in project Gutenberg. Dialogs and narratives written in a modern style are absolutely necessary to train a model that will work well with the way people speak and write today.

This likely involves:

1. Implementing a downloader for [smashwords.com](https://www.smashwords.com/)
2. Implementing an adapter, if necessary, to format the downloaded texts in the way that [gutenberg-dialog](https://github.com/AbrahamSanders/gutenberg-dialog)'s pipeline expects.

https://github.com/soskek/bookcorpus may be a good starting point, as it implements a crawler for smashwords.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.