OpenRefine / OpenRefine/OpenRefine

Support collator strength and decomposition mode for row ordering Sort

Open
#7,431 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

localization Status: Pending Review Type: Feature Request
Dominant language
Java
Stars
12k
Forks
2.2k
Avg merge
1d 14h
Merged PRs (30d)
27

Description

As alluded to in https://github.com/OpenRefine/OpenRefine/issues/6608#issuecomment-2116153415, it would be useful to extend our sorting capabilities to support different strength collators as well as different locales.

We currently support IDENTICAL and SECONDARY strengths, depending on whether the user selects caseSensitive or not

https://github.com/OpenRefine/OpenRefine/blob/6f937d0d8f619f59a12abdd7e5a5b84e4b71128c/modules/core/src/main/java/com/google/refine/sorting/StringCriterion.java#L63

but hard wire the decomposition mode to FULL_DECOMPOSITION and always use the server's default locale (which is likely to match that of the browser in most common, and all supported, cases)

https://github.com/OpenRefine/OpenRefine/blob/6f937d0d8f619f59a12abdd7e5a5b84e4b71128c/modules/core/src/main/java/com/google/refine/sorting/StringCriterion.java#L58

Proposed solution

A complete solution would include giving the user control of:

  • strength - identical, primary, secondary, or tertiary
  • decomposition - none, canonical, full
  • locale - default or selectable from list of any supported

as described in the Java docs. Currently supported options are rendered in bold above.

Alternatives considered

We could support a subset of the above -- or even leave things the way they are.

This only affects the row ordering as done by the Sort function in the UI. It may also be desirable to give the user more fine grained control over the grel sort() function for arrays (although it doesn't even have the basics such as case insensitive sort).

Additional context

Diacritic insensitive collation using the server's default locale was first introduced in #202 in 2012. There was a regression for 3.x, which was fixed when #6047 was resolved for 3.8. The fact that it took several years for the regression to be noticed probably means that most people use, and are happy with, the default case-insensitive, diacritic-sensitive, but natural order, sort using the default locale, perhaps making this low priority.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with modules/core/src/main/java/com/google/refine/sorting/StringCriterion.java, especially the referenced collator configuration, and review the Java Collator documentation. Trace the Sort function's UI entry point to determine where strength, decomposition, and locale choices would be exposed. Done means the supported options affect row ordering without changing unrelated sorting behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, full-stack
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.