clips / clips/pattern

Printing Word & Chunk objects that contain Unicode

Open
#139 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.9k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Hi! I'm working with texts containing Unicode in Python 2.7.9.

I noticed that using "print" on Word objects and Chunk objects seem to behave differently. In short, printing a Word object containing an Unicode character works, but printing a Chunk containing an Unicode character. For example:

Printing: sentence.string
The French for cake is gâteau .

Printing: sentence.words
Word(u'The/DT')
Word(u'French/NNP')
Word(u'for/IN')
Word(u'cake/NN')
Word(u'is/VBZ')
Word(u'g\xe2teau/NN')
Word(u'./.')

Printing: sentence.chunks
Chunk('The French/NP')
Chunk('for/PP')
Chunk('cake/NP-SBJ-1')
Chunk('is/VP-1')
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-21-68762238b21e> in <module>()
     15 print "\nPrinting: sentence.chunks"
     16 for chunk in sentence.chunks:
---> 17     print chunk

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe2' in position 8: ordinal not in range(128)

Maybe using "print" isn't the best way to output Words and Chunks, but it's convenient when experimenting. Is there a reason why Word and Chunk don't behave consistently regarding Unicode?

Thanks :)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Word and Chunk printing examples in Python 2.7.9 and inspect how each object is converted for output. Done means printing a Chunk containing a Unicode character no longer raises UnicodeEncodeError and behaves consistently with Word.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
developer-experience
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.