allenai / allenai/bi-att-flow

nltk tokenize doesn't work?

オープン
#45 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
1.5k
フォーク
668
PR マージ指標
30日以内にマージされた PR はありません

説明

Dear Team,
The code below doesn't work and the context doesn't sententce token.
if args.tokenizer == "PTB":
import nltk
sent_tokenize = nltk.sent_tokenize
def word_tokenize(tokens):
return [token.replace("''", '"').replace("``", '"') for token in nltk.word_tokenize(tokens)]
I check the shared_dev.json, I got this
"x": [
[
[
[
"The",
"income",
"tax",
"withholding",
"rate",
"remains",
"at",
"4.25",
"%",
"for",
"tax",
"year",
"2015",
".",
"However",
",",
"the",
"personal",
"exemption",
"amount",
"for",
"tax",
"year",
"2015",
"will",
"change",
"to",
"$",
"4,000",
".",
"You",
"may",
"continue",
"to",
"use",
"2014",
"Michigan",
"Income",
"Tax",
"Withholding",
"Tables",
"."
],
But, if I change the code like follows, It works.

import nltk.tokenize as nltk
def prepro_each(args, data_type, start_ratio=0.0, stop_ratio=1.0, out_name="default", in_path=None):
if args.tokenizer == "PTB":

# sent_tokenize = nltk.sent_tokenize
def word_tokenize(tokens):
return [token.replace("''", '"').replace("``", '"') for token in nltk.word_tokenize(tokens)]
......
xi = list(map(word_tokenize, nltk.sent_tokenize(context)))

I change the code and run again, but I got a little lower EM and F1. I was very puzzled about it. Could you please help me solve the problem?

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。