nltk tokenize doesn't work?
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 667
- PR merge metrics
- No merged PRs in 30d
Description
Dear Team,
The code below doesn't work and the context doesn't sententce token.
if args.tokenizer == "PTB":
import nltk
sent_tokenize = nltk.sent_tokenize
def word_tokenize(tokens):
return [token.replace("''", '"').replace("``", '"') for token in nltk.word_tokenize(tokens)]
I check the shared_dev.json, I got this
"x": [
[
[
[
"The",
"income",
"tax",
"withholding",
"rate",
"remains",
"at",
"4.25",
"%",
"for",
"tax",
"year",
"2015",
".",
"However",
",",
"the",
"personal",
"exemption",
"amount",
"for",
"tax",
"year",
"2015",
"will",
"change",
"to",
"$",
"4,000",
".",
"You",
"may",
"continue",
"to",
"use",
"2014",
"Michigan",
"Income",
"Tax",
"Withholding",
"Tables",
"."
],
But, if I change the code like follows, It works.
import nltk.tokenize as nltk
def prepro_each(args, data_type, start_ratio=0.0, stop_ratio=1.0, out_name="default", in_path=None):
if args.tokenizer == "PTB":
# sent_tokenize = nltk.sent_tokenize
def word_tokenize(tokens):
return [token.replace("''", '"').replace("``", '"') for token in nltk.word_tokenize(tokens)]
......
xi = list(map(word_tokenize, nltk.sent_tokenize(context)))
I change the code and run again, but I got a little lower EM and F1. I was very puzzled about it. Could you please help me solve the problem?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.