allenai / allenai/bi-att-flow

nltk tokenize doesn't work?

未关闭
#45 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.5k
派生
668
PR 合并指标
30 天内没有已合并 PR

描述

Dear Team,
The code below doesn't work and the context doesn't sententce token.
if args.tokenizer == "PTB":
import nltk
sent_tokenize = nltk.sent_tokenize
def word_tokenize(tokens):
return [token.replace("''", '"').replace("``", '"') for token in nltk.word_tokenize(tokens)]
I check the shared_dev.json, I got this
"x": [
[
[
[
"The",
"income",
"tax",
"withholding",
"rate",
"remains",
"at",
"4.25",
"%",
"for",
"tax",
"year",
"2015",
".",
"However",
",",
"the",
"personal",
"exemption",
"amount",
"for",
"tax",
"year",
"2015",
"will",
"change",
"to",
"$",
"4,000",
".",
"You",
"may",
"continue",
"to",
"use",
"2014",
"Michigan",
"Income",
"Tax",
"Withholding",
"Tables",
"."
],
But, if I change the code like follows, It works.

import nltk.tokenize as nltk
def prepro_each(args, data_type, start_ratio=0.0, stop_ratio=1.0, out_name="default", in_path=None):
if args.tokenizer == "PTB":

# sent_tokenize = nltk.sent_tokenize
def word_tokenize(tokens):
return [token.replace("''", '"').replace("``", '"') for token in nltk.word_tokenize(tokens)]
......
xi = list(map(word_tokenize, nltk.sent_tokenize(context)))

I change the code and run again, but I got a little lower EM and F1. I was very puzzled about it. Could you please help me solve the problem?

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。