[Java] Unable to update VectorSchemaRoot field vectors.
- Langage dominant
- Java
- Étoiles
- 94
- Forks
- 152
- Merge moyen
- 3 j 16 h
- PR mergées (30 j)
- 11
Description
### Describe the usage question you have. Please include as many useful details as possible.
Hi community,
I have a trying to modify data buffers of string column.
if i print the buffer readable bytes after ` buffers.set(index, compressedBuf); `, i see the right value.
```
List buffers = fieldVector.getFieldBuffers();
int index = buffers.size() - 1;
ArrowBuf compressedBuf = null;
ArrowBuf originalBuf = buffers.get(index);
ByteBuffer originalBuffer = originalBuf.nioBuffer(0, (int) originalBuf.readableBytes());
// Compress the buffer using LZ4
byte[] originalBytes = new byte[originalBuffer.remaining()];
System.out.println("This is buffer id: "+ index + " originalbytes: "+ originalBuf.readableBytes());
originalBuffer.get(originalBytes);
int maxCompressedLength = compressor.maxCompressedLength(originalBytes.length);
byte[] compressedBytes = new byte[maxCompressedLength];
int compressedLength = compressor.compress(originalBytes, 0, originalBytes.length, compressedBytes, 0, maxCompressedLength);
// Allocate a new ArrowBuf for the compressed data
compressedBuf = allocator.buffer(compressedLength);
compressedBuf.writeBytes(compressedBytes, 0, compressedLength);
compressedBuf.writerIndex(compressedLength);
System.out.println("This is buffer id: "+ index+ " compressedbytes: "+ compressedBuf.readableBytes());
buffers.set(index, compressedBuf); // Replace with compressed buffer
```
When i update my fieldvector.
```
int nullCount = fieldVector.getNullCount(); // Get the null count
ArrowFieldNode fieldNode = new ArrowFieldNode(fieldVectorValueCount, nullCount);
ArrowBuf validityBuffer = fieldVector.getValidityBuffer(); // Assuming it exists
ArrowBuf offsetBuffer = fieldVector.getOffsetBuffer(); // Assuming it exists
// Update FieldVector with the new compressed buffers
fieldVector.clear(); // Clear existing buffers (to prevent memory leaks)
fieldVector.loadFieldBuffers(fieldNode,java.util.Arrays.asList(validityBuffer, offsetBuffer, compressedBuf)); // Load the new (compressed) buffers
fieldVector.setValueCount(fieldVectorValueCount); // Reset value count after buffer replacement
Field field1 = fieldVector.getField();
```
and i check the buffers again then buffers reflect old value.
please advise .
### Component(s)
Java
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Examinez les API de VectorSchemaRoot et FieldVector impliquées dans getFieldBuffers(), clear() et loadFieldBuffers(), en commençant par leurs contrats de propriété et de remplacement des buffers. Reproduisez la séquence présentée et vérifiez si le vecteur obtenu expose compressedBuf ou les buffers d’origine ; c’est terminé lorsqu’un chemin de mise à jour pris en charge est identifié ou que le comportement est confirmé par un test ciblé.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java
- Domaine
- data-engineering
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- À clarifier
- Accessibilité débutants
- 25/100