EleutherAI / EleutherAI/dps

Need to add ignore null or empty text during korean text process

Open
#56 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
93
Forks
26
PR merge metrics
No merged PRs in 30d

Description

### Agenda

- Some raw datasets can have empty or null text.
- Using `filter` method in RDD or DF, text like `""` need to be ignored during this process.

### How to solve this problem?

- Add `filter` method after load the data
- Add condition if the text data is null or empty in filter method

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.