[C Data] Format.asString emits locale dependent digits in format strings
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
Research direction
Start in c/src/main/java/org/apache/arrow/c/Format.java at the String.format calls linked in the issue, then run the provided reproduction with the ar-EG locale. Done means Format.asString emits ASCII digits for FixedSizeList, FixedSizeBinary, and Decimal parameters, while the existing Java import behavior remains compatible.
Written by the indexing model from the issue text.
Description
Describe the bug, including details regarding any error messages, version, and platform.
Format.asString builds the parameterised format strings with String.format("+w:%d", ...) and no explicit Locale. java.util.Formatter localises the digits of %d with Locale.getDefault(Locale.Category.FORMAT), so on a JVM whose default locale uses digits outside ASCII (ar-EG, fa-IR, bn-BD, mr-IN, 98 of the 1069 locales available on JDK 21) a FixedSizeList(8) is exported as +w:٨ instead of +w:8. The same applies to FixedSizeBinary and Decimal.
https://github.com/apache/arrow-java/blob/b410fb26d01c7cefc4c6e3443a53562164001371/c/src/main/java/org/apache/arrow/c/Format.java#L66
https://github.com/apache/arrow-java/blob/b410fb26d01c7cefc4c6e3443a53562164001371/c/src/main/java/org/apache/arrow/c/Format.java#L68-L69
https://github.com/apache/arrow-java/blob/b410fb26d01c7cefc4c6e3443a53562164001371/c/src/main/java/org/apache/arrow/c/Format.java#L96
https://github.com/apache/arrow-java/blob/b410fb26d01c7cefc4c6e3443a53562164001371/c/src/main/java/org/apache/arrow/c/Format.java#L91
Other implementations parse the parameter as an ASCII integer and reject the string. arrow-rs fails at https://github.com/apache/arrow-rs/blob/58.0.0/arrow-schema/src/ffi.rs#L494-L503 with C Data interface error: The FixedSizeList type requires an integer parameter representing number of elements per list. The Java importer reads the parameter with Integer.parseInt (https://github.com/apache/arrow-java/blob/v19.0.0/c/src/main/java/org/apache/arrow/c/Format.java#L352-L354), which accepts Unicode digits, so a Java to Java round trip does not expose the problem.
Reproduction (arrow-java 19.0.0 and 18.1.0, Temurin 21.0.6, run with --add-opens=java.base/java.nio=ALL-UNNAMED):
import java.nio.charset.StandardCharsets;
import java.util.HexFormat;
import java.util.List;
import java.util.Locale;
import org.apache.arrow.c.ArrowSchema;
import org.apache.arrow.c.Data;
import org.apache.arrow.c.NativeUtil;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.types.FloatingPointPrecision;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.FieldType;
public class FormatLocale {
public static void main(String[] args) {
Locale.setDefault(Locale.forLanguageTag("ar-EG"));
Field item = new Field("item",
FieldType.nullable(new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE)), null);
Field vec = new Field("vec", FieldType.nullable(new ArrowType.FixedSizeList(8)), List.of(item));
try (BufferAllocator allocator = new RootAllocator()) {
ArrowSchema schema = ArrowSchema.allocateNew(allocator);
Data.exportField(allocator, vec, null, schema);
String format = NativeUtil.toJavaString(schema.snapshot().format);
System.out.println(format + " "
+ HexFormat.ofDelimiter(" ").formatHex(format.getBytes(StandardCharsets.UTF_8)));
System.out.println("Java import: " + Data.importField(allocator, schema, null).getType());
}
}
}
+w:٨ 2b 77 3a d9 a8
Java import: FixedSizeList(8)
With Locale.US the first line is +w:8 2b 77 3a 38.
- Dominant language
- Java
- Stars
- 95
- Forks
- 154
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 9
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/arrow-java
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
apache/arrow-java#1261 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/arrow-java#1236 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/arrow-java#1230 ·
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
apache/arrow-java#1205 ·
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/arrow-java#1196 · 1 comment ·
All issues in apache/arrow-java
Similar issues
-
Bug Java Platform: Java
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
getsentry/sentry-java#6138 · 1 comment ·
-
bug needs triage p2
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
GoogleCloudPlatform/DataflowTemplates#4273 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
[Studio][Bug] Bulk-deleting a full page of alert rules steps the page back while more rules remain Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/rocketmq-dashboard#4654 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100