FasterXML / FasterXML/jackson-dataformats-binary

Invalid Avro file produced using SequenceWriter

Aperta
#339 7 commenti 0 reazioni 0 assegnatari Vedi su GitHub
avro
Lingua principale
Java
Stelle
347
Fork
156
Merge medio
3g 3h
PR unite (30g)
22

Descrizione

While documentation on writing Avro to a file is sparse, I have managed to piece some stuff together but I am still getting an error.

- https://groups.google.com/g/jackson-user/c/yOXaJvAMzfg
- https://github.com/FasterXML/jackson-dataformats-binary/issues/35
- https://github.com/FasterXML/jackson-dataformats-binary/issues/15

Here is some sample code:

```java
final var avroFactory = AvroFactory.builderWithApacheDecoder().enable(AvroGenerator.Feature.AVRO_FILE_OUTPUT).build();

final var generator = new AvroSchemaGenerator().enableLogicalTypes();

final var mapper = AvroMapper.builder(avroFactory).addModule(new AvroJavaTimeModule()).build();
mapper.acceptJsonFormatVisitor(Thing.class, generator);

final var avroSchema = generator.getGeneratedSchema();

final var file = Files.createTempFile("something", ".avro").toFile();

final var out = new ByteArrayOutputStream();
final var writer = mapper.writer(avroSchema).writeValues(out);

// in a loop
writer.write(thing);

// after loop
writer.close();

try (FileOutputStream outputStream = new FileOutputStream(file)) {
out.writeTo(outputStream);
}
```

When checking the resultant file using [`avro-tools`](https://avro.apache.org/), I get the following error:

```
avro-tools tojson something.avro

22/09/08 18:36:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException: Invalid sync!
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:224)
at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:97)
at org.apache.avro.tool.Main.run(Main.java:67)
at org.apache.avro.tool.Main.main(Main.java:56)
Caused by: java.io.IOException: Invalid sync!
at org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:319)
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:213)
... 3 mor
```

According to some searching, the `Invalid sync!` error occurs when the file hasn't been stitched together properly, but it's unclear to me what I need to do in code to get that to happen. I've looked through most of the Avro tests in this repo and I cannot find one that actually writes to a file and then de-serializes from that file.

I am not sure if I have stumbled into an actual bug here or not, but I am happy to try and write a test case if this code does seem correct since that would imply it's a bug?

Thanks in advance.

Edit:

I've also tried the following:

```java
final var file = Files.createTempFile("something", ".avro").toFile();
final SequenceWriter writer = mapper.writer(avroSchema).writeValues(file);
```

In which case I get the following error at that line:

```
java.lang.UnsupportedOperationException: Generator of type com.fasterxml.jackson.core.json.UTF8JsonGenerator does not support schema of type 'avro'

at com.fasterxml.jackson.core.JsonGenerator.setSchema(JsonGenerator.java:592)
at com.fasterxml.jackson.databind.ObjectWriter$GeneratorSettings.initialize(ObjectWriter.java:1393)
at com.fasterxml.jackson.databind.ObjectWriter._configureGenerator(ObjectWriter.java:1258)
at com.fasterxml.jackson.databind.ObjectWriter.createGenerator(ObjectWriter.java:717)
at com.fasterxml.jackson.databind.ObjectWriter.writeValues(ObjectWriter.java:753)
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.