amzn / amzn/pecos

Newlines in Tfidf vectorizer corpus cause runtime exceptions when loading a trained vectorizer

Open
#263 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
550
Forks
113
Avg merge
21h 37m
Merged PRs (30d)
1

Description

## Description
When using the [Pecos Tfidf Vectorizer](https://github.com/amzn/pecos/blob/mainline/pecos/utils/featurization/text/vectorizers.py), if you train it using a corpus which includes newlines, this causes errors when loading the saved version. The error is because the vocab file is parsed using newlines to [delimit (index,vocab) pairs](https://github.com/amzn/pecos/blob/mainline/pecos/core/utils/tfidf.hpp#L363-L386) and if the vocab contains a newline it will crash since the entry is now across multiple lines.

## How to Reproduce?
Using latest version of libpecos

### Steps to reproduce

```python
Python 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:23:14) [GCC 10.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pecos.utils.featurization.text.vectorizers import Tfidf
>>> vectorizer = Tfidf()
>>> trained = vectorizer.train(["test\ncorpus"], config={'ngram_range':(1,1)})
>>> trained.save('test')
>>> Tfidf.load('test')
terminate called after throwing an instance of 'std::runtime_error'
what(): Corrupted vocab file.
Aborted
```

## What have you tried to solve it?

1. This is solvable by cleaning the input but it may be desirable to handle this case internally so that cases where newlines are important do not break the vectorizer.

## Error message or code output
(Paste the complete error message, including stack trace, or the undesired output that the above snippet produces.)

```
terminate called after throwing an instance of 'std::runtime_error'
what(): Corrupted vocab file.
Aborted
```

## Environment
- Operating system:
- Python version: 3.10
- PECOS version: 1.2

Contributor guide

Open the contributing guide

Research direction

Inspect pecos/utils/featurization/text/vectorizers.py and the vocab parsing code in pecos/core/utils/tfidf.hpp at lines 363-386. Reproduce the shown save/load case with a newline-containing corpus, then verify that the trained vectorizer can be saved and loaded without the “Corrupted vocab file” exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.