apache / apache/arrow-java

[Java] DictionaryEncoder doesn't crash when decoding index outside of Dictionary

オープン 初心者向け
#1,158 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
Type: bug
主要言語
Java
スター
94
フォーク
152
平均マージ
3日 16時間
マージ済み PR(30日)
11

説明

### Describe the bug, including details regarding any error messages, version, and platform.

### Background
When manually creating an encoded vector, with values referencing indices in a Dictionary, it is possible to have values equal to `valueCount` of the Dictionary vector i.e. one index out of bounds. This is interpreted as a null value in the Dictionary.

---

### Example
_Dictionary_
| Index | Original Value |
|-------|---------------|
| 0 | Foo |

_Encoded Vector - 1_
| Index | Encoded Value | Expected Decode Outcome | Actual Outcome |
|-------|----------------|-----------------------------------|-----------------|
| 0 | 0 | Return the original value | Returns the original value |

_Encoded Vector - 2_
| Index | Encoded Value | Expected Decode Outcome | Actual Outcome |
|-------|----------------|-----------------------------------|-----------------|
| 0 | 1 | Raise `IllegalArgumentException` | Returns null |

_Encoded Vector - 3_
| Index | Encoded Value | Expected Decode Outcome | Actual Outcome |
|-------|----------------|-----------------------------------|-----------------|
| 0 | 2 | Raise `IllegalArgumentException` | Raises `IllegalArgumentException`|

---

### Test to reproduce the error
```java
@Test
public void testReferencingIndexOutOfBounds() {
// Index at which the original value will be stored at in the dictionary
var encodedIndex = 0;
// The encoded value that references an index in the dictionary
var indexReferenced = 1;

try (final IntVector encodedVector = new IntVector("encodings", allocator);
final VarCharVector dictionaryVector = newVarCharVector("dict", allocator); ) {
var originalValue = "Foo";
dictionaryVector.allocateNew(1);
dictionaryVector.setValueCount(1);
dictionaryVector.set(encodedIndex, originalValue.getBytes(StandardCharsets.UTF_8));

encodedVector.allocateNew(1);
encodedVector.setValueCount(1);
encodedVector.set(0, indexReferenced);

Dictionary dictionary =
new Dictionary(dictionaryVector, new DictionaryEncoding(1L, false, null));

try (ValueVector decoded = DictionaryEncoder.decode(encodedVector, dictionary)) {
fail("There should be an exception when decoding index outside dictionary's range.");
} catch (Exception e) {
assertEquals("Provided dictionary does not contain value for index " + indexReferenced, e.getMessage());
}
}
}

```

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start at DictionaryEncoder.decode and reproduce the behavior with the provided testReferencingIndexOutOfBounds test. Confirm that an encoded index equal to the dictionary value count raises IllegalArgumentException with the expected message, while valid indices still decode normally.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
data
issue の種類
バグ
難易度
2/5
見積もり時間
1〜3時間
活発さ
静か
明瞭さ
明確に書かれている
初心者へのやさしさ
74/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。