selfteaching / selfteaching/selfteaching-python-camp

Day10 关于jieba输入返回的数据类型——从re筛选到cut分词再到counter统计的数据类型衔接

Open
#1,273 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
151
Forks
875
PR merge metrics
No merged PRs in 30d

Description

在使用jieba进行分词时,我遇到了两个难点。
1、cut语句返回的是一个生成器,不是字符串或是列表;
2、使用分词功能时,jieba接收的是字符串类型,但是我在筛选汉字时返回的是列表。
具体描述和解决方法如下:(不是唯一正确,仅供参考)

1、cut语句返回的是一个生成器,不是字符串或是列表
jieba的github网页里readme.md文件里的代码示例如下:

seg_list = jieba.cut("我来到北京清华大学", cut_all=False)

这个语句分词后返回的是一个生成器,这个类型在衔接Counter做词频统计时,因为Counter要接收列表类型(list),所以运行时会有 AttributeError: 'list' object has no attribute 'decode' 这样的错误报告。解决办法是把语句中cut改为lcut,这样返回的就是列表类型(list)了,可以直接用于Counter。示例如下:

seg_list = jieba.lcut("我来到北京清华大学", cut_all=False)

2、使用分词功能时,jieba接收的是字符串类型,但是我在筛选汉字时返回的是列表
我在区别中英文时,用的筛选方法是:

import re
cn = re.compile(r'[\u4e00-\u9fa5]')
text_cn = re.findall(cn, text)

但是因为jieba分词功能需要接收字符串类型(str),而上面语句返回的是一个列表类型(list)
,所以会报错 AttributeError: 'list' object has no attribute 'decode' ,解决办法是把re筛选返回的(list)改为(str)再衔接到jieba的lcut语句,示例如下:

import re
cn = re.compile(r'[\u4e00-\u9fa5]')
text_cn = re.findall(cn, text)
text_cut = ''.join(text_cn)
cut_list = jieba.lcut(text_cut,cut_all=False)
count_list = Counter(cut_list).most_common(20)

这样几个从筛选出中文到jieba分词再到Counter统计词频就能衔接了。

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue references README.md examples for jieba.cut/lcut and the re/Counter snippets, but names no repository file or requested edit. Start by locating the relevant lesson or README section and verify the documented types and error message. Done means the intended Python data flow and any needed clarification are documented accurately.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.