[C Data] Format.asString emits locale dependent digits in format strings

Open Beginner friendly
#1,300 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
java
Domain
data

Research direction

Start in c/src/main/java/org/apache/arrow/c/Format.java at the String.format calls linked in the issue, then run the provided reproduction with the ar-EG locale. Done means Format.asString emits ASCII digits for FixedSizeList, FixedSizeBinary, and Decimal parameters, while the existing Java import behavior remains compatible.

Written by the indexing model from the issue text.

Description

Type: bug
Describe the bug, including details regarding any error messages, version, and platform.

Format.asString builds the parameterised format strings with String.format("+w:%d", ...) and no explicit Locale. java.util.Formatter localises the digits of %d with Locale.getDefault(Locale.Category.FORMAT), so on a JVM whose default locale uses digits outside ASCII (ar-EG, fa-IR, bn-BD, mr-IN, 98 of the 1069 locales available on JDK 21) a FixedSizeList(8) is exported as +w:٨ instead of +w:8. The same applies to FixedSizeBinary and Decimal.

https://github.com/apache/arrow-java/blob/b410fb26d01c7cefc4c6e3443a53562164001371/c/src/main/java/org/apache/arrow/c/Format.java#L66
https://github.com/apache/arrow-java/blob/b410fb26d01c7cefc4c6e3443a53562164001371/c/src/main/java/org/apache/arrow/c/Format.java#L68-L69
https://github.com/apache/arrow-java/blob/b410fb26d01c7cefc4c6e3443a53562164001371/c/src/main/java/org/apache/arrow/c/Format.java#L96
https://github.com/apache/arrow-java/blob/b410fb26d01c7cefc4c6e3443a53562164001371/c/src/main/java/org/apache/arrow/c/Format.java#L91

Other implementations parse the parameter as an ASCII integer and reject the string. arrow-rs fails at https://github.com/apache/arrow-rs/blob/58.0.0/arrow-schema/src/ffi.rs#L494-L503 with C Data interface error: The FixedSizeList type requires an integer parameter representing number of elements per list. The Java importer reads the parameter with Integer.parseInt (https://github.com/apache/arrow-java/blob/v19.0.0/c/src/main/java/org/apache/arrow/c/Format.java#L352-L354), which accepts Unicode digits, so a Java to Java round trip does not expose the problem.

Reproduction (arrow-java 19.0.0 and 18.1.0, Temurin 21.0.6, run with --add-opens=java.base/java.nio=ALL-UNNAMED):

import java.nio.charset.StandardCharsets;
import java.util.HexFormat;
import java.util.List;
import java.util.Locale;
import org.apache.arrow.c.ArrowSchema;
import org.apache.arrow.c.Data;
import org.apache.arrow.c.NativeUtil;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.types.FloatingPointPrecision;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.FieldType;

public class FormatLocale {
  public static void main(String[] args) {
    Locale.setDefault(Locale.forLanguageTag("ar-EG"));
    Field item = new Field("item",
        FieldType.nullable(new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE)), null);
    Field vec = new Field("vec", FieldType.nullable(new ArrowType.FixedSizeList(8)), List.of(item));
    try (BufferAllocator allocator = new RootAllocator()) {
      ArrowSchema schema = ArrowSchema.allocateNew(allocator);
      Data.exportField(allocator, vec, null, schema);
      String format = NativeUtil.toJavaString(schema.snapshot().format);
      System.out.println(format + "  "
          + HexFormat.ofDelimiter(" ").formatHex(format.getBytes(StandardCharsets.UTF_8)));
      System.out.println("Java import: " + Data.importField(allocator, schema, null).getType());
    }
  }
}
+w:٨  2b 77 3a d9 a8
Java import: FixedSizeList(8)

With Locale.US the first line is +w:8 2b 77 3a 38.

Dominant language
Java
Stars
95
Forks
154
Avg merge
2d 16h
Merged PRs (30d)
9

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/arrow-java

All issues in apache/arrow-java

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.