Diacritics are not considered part of words

Open
#101,421 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
backend

Research direction

The report provides a Python re and unicodedata reproduction but names no CPython file or test. Start by locating the Unicode handling for \w in the regex engine and reviewing related tests; done requires an agreed behavior for decomposed diacritics and a regression test.

Written by the indexing model from the issue text.

Description

stdlib topic-regex topic-unicode type-bug

I'm not sure whether it's a bug or expected behaviour, but it seems odd so I figure reporting it is a good idea: while a precomposed character is considered "a word" by the regex engine (specifically \w), its decomposed form is not, because a diacritic is not considered part of a word.

>>> import re, unicodedata
>>> s = "ö"
>>> list(s)
['ö']
>>> list(unicodedata.normalize('NFD', s))
['o', '̈']
>>> re.fullmatch(r'\w+', s)
<re.Match object; span=(0, 1), match='ö'>
>>> re.fullmatch(r'\w+', unicodedata.normalize('NFD', s))

This leads to odd effects when ingesting and filtering decomposed data.

Tested on 3.8.13, 3.10.6, and 3.11.1 (all installed via pyenv), on a Mint 21.1).

Dominant language
Python
Stars
77.2k
Forks
36k
Avg merge
1d 9h
Merged PRs (30d)
558

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from python/cpython

All issues in python/cpython

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.