apache / apache/paimon

[Bug] Locale-sensitive case conversions break identifier matching, enum parsing and request signing

Open
#9,770 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Bug description

`String.toLowerCase()` and `String.toUpperCase()` follow the JVM default locale. Under a Turkish or Azeri default, `i` uppercases to the dotted `İ` and `I` lowercases to the dotless `ı`, so any token that is case-folded before being matched, parsed or signed stops matching what it is compared against.

This started as an identifier-matching bug. With `case-sensitive = false`, CDC table mapping, computed columns and the Arrow readers all route through `StringUtils.toLowerCaseIfNeed`, and `CdcRecord.fieldNameLowerCase` is the record side of the same join, so a source column `INDEX` stops mapping to schema field `index` and the column silently nulls out.

Auditing the rest of `paimon-api`, `paimon-common` and `paimon-core` turned up the same conversion in places where it throws rather than mismatching quietly:

- `CoreOptions.partitionMarkDoneActions()` does `PartitionMarkDoneAction.valueOf(x.replace('-','_').toUpperCase())`. Both `success-file` and `done-partition` contain an `i`, so under Turkish this throws `IllegalArgumentException: No enum constant ...PartitionMarkDoneAction.SUCCESS_FİLE`.
- `RowKind.fromShortString("+i")` uppercases to `+İ`, matches no case arm, and throws `UnsupportedOperationException`.
- `DistributedLockDialectFactory` does `JdbcProtocol.valueOf(protocol.toUpperCase())` and loses `SQLITE` and `MARIADB`, so JDBC catalog locking cannot resolve its dialect.
- `OperatingSystem` lowercases `os.name` and then looks for `solaris`, which becomes `solarıs`, so the OS is reported as `UNKNOWN`.
- The DLF request signers lowercase header names to build the signature, option key lookups and format identifiers lowercase user-supplied keys, and system table lookup lowercases the table name. All of these compare the result against a fixed ASCII string.

### Expected behavior

None of these conversions should depend on the operator's locale: they are all machine-facing, so they should pin `Locale.ROOT`.

`BinaryString.toLowerCase` / `toUpperCase` are already safe: the ASCII path uses `Character.toLowerCase` and the non-ASCII fallback pins `Locale.ROOT`, so the SQL `upper()` / `lower()` transforms over user data are not affected.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by auditing locale-sensitive conversions in paimon-api, paimon-common, and paimon-core, focusing on StringUtils.toLowerCaseIfNeed, CdcRecord.fieldNameLowerCase, CoreOptions.partitionMarkDoneActions(), RowKind.fromShortString(), DistributedLockDialectFactory, OperatingSystem, and the DLF request signers. Verify each machine-facing comparison uses Locale.ROOT while leaving BinaryString conversions unchanged. Run the relevant existing tests and confirm identifier matching, enum parsing, dialect resolution, OS detection, and request signing work under Turkish or Azeri defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.